Sep 17 23:09:20.918984 ip-10-0-128-184 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 23:09:20.919094 ip-10-0-128-184 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 23:09:20.919103 ip-10-0-128-184 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 23:09:20.919433 ip-10-0-128-184 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 23:09:31.111515 ip-10-0-128-184 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 23:09:31.111538 ip-10-0-128-184 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2ca174e0535040bcb651db29adf5f389 --
Sep 17 23:11:06.851910 ip-10-0-128-184 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 23:11:07.280819 ip-10-0-128-184 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:11:07.280819 ip-10-0-128-184 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 23:11:07.281287 ip-10-0-128-184 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:11:07.281287 ip-10-0-128-184 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 23:11:07.281287 ip-10-0-128-184 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:11:07.281968 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.281890    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 23:11:07.287419 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287399    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:11:07.287419 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287417    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:11:07.287419 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287424    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287429    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287432    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287435    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287438    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287440    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287443    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287446    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287448    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287451    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287453    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287456    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287458    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287461    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287463    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287466    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287469    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287471    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287473    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287476    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:11:07.287553 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287478    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287481    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287486    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287491    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287495    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287498    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287500    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287503    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287506    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287510    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287513    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287516    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287518    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287522    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287524    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287527    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287530    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287532    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287534    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:11:07.288038 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287537    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287540    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287543    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287545    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287547    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287550    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287552    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287555    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287558    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287560    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287563    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287565    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287568    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287570    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287572    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287575    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287579    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287581    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287584    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287586    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287589    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:11:07.288481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287591    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287593    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287595    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287598    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287600    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287603    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287606    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287628    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287631    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287635    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287637    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287640    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287642    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287645    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287647    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287649    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287652    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287654    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287657    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:11:07.289079 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287659    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287662    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287665    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287667    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287670    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287672    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287674    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287677    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287679    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287682    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287684    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287686    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287688    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287691    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287693    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287695    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287698    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.287700    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288353    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288360    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:11:07.289534 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288363    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288365    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288368    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288372    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288375    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288377    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288380    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288382    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288385    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288387    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288390    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288392    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288395    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288397    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288400    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288403    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288405    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288408    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288410    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288412    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:11:07.290031 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288415    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288418    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288420    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288422    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288425    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288427    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288429    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288432    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288434    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288436    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288439    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288443    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288446    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288449    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288451    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288453    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288456    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288458    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288461    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:11:07.290515 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288464    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288466    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288469    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288473    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288477    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288480    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288483    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288486    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288488    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288491    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288496    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288499    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288501    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288504    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288506    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288509    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288511    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288514    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288517    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:11:07.290991 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288519    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288521    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288524    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288526    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288531    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288534    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288536    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288538    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288541    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288543    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288545    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288547    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288550    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288552    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288555    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288557    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288559    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288562    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288566    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288569    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:11:07.291428 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288571    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288574    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288577    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288580    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288582    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288585    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288587    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288590    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288592    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288595    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288598    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288600    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288603    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288605    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288607    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288630    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288634    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288637    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288639    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.288998    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:11:07.291920 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289003    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289006    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289009    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289011    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289014    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289017    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289019    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289022    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289024    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289027    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289029    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289032    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289034    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289037    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289039    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289042    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289045    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289047    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289049    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:11:07.292390 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289052    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289054    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289057    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289060    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289062    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289064    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289067    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289070    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289073    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289077    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289081    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289085    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289088    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289091    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289094    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289096    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289099    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289101    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289104    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:11:07.292878 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289106    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289109    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289111    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289113    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289116    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289118    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289121    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289123    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289125    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289128    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289131    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289133    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289136    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289138    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289141    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289144    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289147    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289149    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289151    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289154    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:11:07.293334 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289156    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289158    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289162    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289165    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289167    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289170    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289174    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289176    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289178    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289181    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289183    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289185    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289188    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289190    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289192    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289195    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289197    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289199    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289202    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289205    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:11:07.293859 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289208    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289211    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289213    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289216    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289218    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289220    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289223    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289226    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289230    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289234    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289236    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289239    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289241    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289243    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289247    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289250    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289253    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289255    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289257    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289260    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:11:07.294331 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289328    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289365    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289372    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289376    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289380    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289383    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289387    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289391    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289394    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289397    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289400    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289403    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289406    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289409    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289411    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289414    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289417    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289420    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289423    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289427    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289430    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289433    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289436    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289440    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 23:11:07.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289442    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289445    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289449    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289452    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289463    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289466    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289469    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289472    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289476    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289479    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289481    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289484    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289487    2580 flags.go:64] FLAG: --enable-server="true"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289489    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289497    2580 flags.go:64] FLAG: --event-burst="100"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289500    2580 flags.go:64] FLAG: --event-qps="50"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289503    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289506    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289508    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289512    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289515    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289517    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289520    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289522    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289525    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 23:11:07.295386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289528    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289531    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289536    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289539    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289542    2580 flags.go:64] FLAG: --feature-gates=""
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289546    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289548    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289551    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289554    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289558    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289561    2580 flags.go:64] FLAG: --help="false"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289564    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289566    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289569    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289572    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289575    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289578    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289580    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289583    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289586    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289588    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289591    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289593    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289596    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 17 23:11:07.296008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289598    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289601    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289604    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289606    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289626    2580 flags.go:64] FLAG: --lock-file=""
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289629    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289632    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289635    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289647    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289651    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289654    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289657    2580 flags.go:64] FLAG: --logging-format="text"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289659    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289662    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289665    2580 flags.go:64] FLAG: --manifest-url=""
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289667    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289671    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289675    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289678    2580 flags.go:64] FLAG: --max-pods="110"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289681    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289684    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289687    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289689    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289692    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289695    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 23:11:07.296583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289698    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289704    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289707    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289710    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289713    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289715    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289720    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289723    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289725    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289728    2580 flags.go:64] FLAG: --port="10250"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289731    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289733    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e5cb40578ddb0f2b"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289736    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289739    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289742    2580 flags.go:64] FLAG: --register-node="true"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289744    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289748    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289751    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289754    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289756    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289759    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289762    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289765    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289768    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289770    2580 flags.go:64] FLAG: --runonce="false"
Sep 17 23:11:07.297191 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289773    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289776    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289779    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289781    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289784    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289787    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289790    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289792    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289795    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289798    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289800    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289803    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289805    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289808    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289810    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289815    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289817    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289820    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289824    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289826    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289829    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289832    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289834    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289837    2580 flags.go:64] FLAG: --v="2"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289841    2580 flags.go:64] FLAG: --version="false"
Sep 17 23:11:07.297803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289845    2580 flags.go:64] FLAG: --vmodule=""
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289848    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.289851    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289960    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289963    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289966    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289968    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289971    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289973    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289976    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289978    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289981    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289983    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289986    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289988    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289990    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289993    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289995    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.289998    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290000    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:11:07.298391 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290002    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290005    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290007    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290009    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290012    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290014    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290017    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290019    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290022    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290024    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290027    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290030    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290032    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290036    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290039    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290041    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290044    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290046    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290050    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:11:07.298906 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290052    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290055    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290057    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290060    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290062    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290064    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290067    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290069    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290072    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290074    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290076    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290078    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290082    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290084    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290087    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290090    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290093    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290095    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290099    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290103    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:11:07.299355 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290106    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290110    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290113    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290115    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290118    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290121    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290123    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290126    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290128    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290131    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290133    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290137    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290139    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290142    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290144    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290148    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290151    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290154    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290156    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:11:07.299851 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290159    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290161    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290163    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290166    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290168    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290171    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290173    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290176    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290178    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290182    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290184    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290187    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290189    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290192    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290196    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290199    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290202    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290204    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290207    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:11:07.300296 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290209    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290212    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290214    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290217    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.290219    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.291007    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.298747    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.298762    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.298775    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.298781    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298833    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298837    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298840    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298844    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:11:07.300748 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298847    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298850    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298852    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298855    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298858    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298860    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298863    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298865    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298868    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298870    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298873    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298875    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298878    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298880    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298883    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298885    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298888    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298891    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298893    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298895    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:11:07.301087 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298898    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298901    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298903    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298906    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298908    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298912    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298915    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298918    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298921    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298923    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298926    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298928    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298931    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298933    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298936    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298938    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298941    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298944    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298946    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:11:07.301566 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298948    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298951    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298955    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298959    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298962    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298965    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298967    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298970    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298973    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298976    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298978    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298980    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298984    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298987    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298990    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298993    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298995    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.298998    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299002    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:11:07.302066 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299005    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299008    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299010    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299013    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299015    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299018    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299021    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299023    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299025    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299029    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299033    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299035    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299038    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299040    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299043    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299045    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299048    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299050    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299053    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299055    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:11:07.302508 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299057    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299060    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299062    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299064    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299067    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299069    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299072    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299074    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299076    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299079    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299081    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299084    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299086    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299089    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299091    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299094    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:11:07.302988 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299097    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.299101    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299208    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299213    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299215    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299218    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299221    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299223    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299226    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299228    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299231    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299233    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299236    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299240    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299242    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:11:07.303359 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299245    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299247    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299250    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299252    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299255    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299257    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299260    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299263    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299265    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299268    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299270    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299273    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299276    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299279    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299282    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299286    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299288    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299291    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299294    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:11:07.303729 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299296    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299299    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299302    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299304    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299306    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299309    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299311    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299313    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299316    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299318    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299320    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299323    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299325    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299327    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299330    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299332    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299334    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299337    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299339    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299342    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:11:07.304163 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299344    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299346    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299349    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299351    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299354    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299356    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299359    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299361    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299364    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299367    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299370    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299372    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299374    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299377    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299379    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299381    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299384    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299386    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299389    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299391    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:11:07.304636 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299393    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299396    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299398    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299400    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299403    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299405    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299407    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299410    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299412    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299415    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299417    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299420    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299422    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299424    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299427    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299429    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299431    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299434    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299436    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:11:07.305099 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299439    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299441    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299443    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299446    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299448    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299452    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299455    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:07.299458    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.299463    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.299554    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.303392    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.304455    2580 server.go:1019] "Starting client certificate rotation"
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.304548    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:11:07.305546 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.304577    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:11:07.307234 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.307222    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:11:07.310604 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.310582    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:11:07.322843 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.322824    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 17 23:11:07.324557 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.324543    2580 log.go:25] "Validated CRI v1 image API"
Sep 17 23:11:07.325270 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.325255    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 23:11:07.327830 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.327810    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7f0556b3-61bf-4306-a507-9143867e845b:/dev/nvme0n1p3 b76740bf-3073-4e27-9e18-771b1f9f437b:/dev/nvme0n1p4]
Sep 17 23:11:07.327902 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.327830    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 23:11:07.332565 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.332548    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:11:07.332780 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.332676    2580 manager.go:216] Machine: {Timestamp:2026-09-17 23:11:07.331294093 +0000 UTC m=+0.375670244 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201222 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e0c40fb0d21662a046ab8acaa8564 SystemUUID:ec2e0c40-fb0d-2166-2a04-6ab8acaa8564 BootID:2ca174e0-5350-40bc-b651-db29adf5f389 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:6e:b8:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:6e:b8:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:f0:8a:b1:38:35 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 23:11:07.332780 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.332777    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 23:11:07.332928 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.332913    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 23:11:07.336221 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.336194    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 23:11:07.336379 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.336223    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-184.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 23:11:07.337800 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.337788    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 23:11:07.337857 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.337805    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 23:11:07.337857 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.337815    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:11:07.338554 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.338543    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:11:07.340534 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.340523    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 23:11:07.345211 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.345198    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:11:07.345338 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.345327    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 23:11:07.345391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.345362    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 23:11:07.345391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.345375    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 23:11:07.345482 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.345394    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 23:11:07.345482 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.345407    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 17 23:11:07.345482 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.345419    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 23:11:07.345794 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.345774    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:11:07.345854 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.345805    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:11:07.348154 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.348136    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 23:11:07.350985 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.350972    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 23:11:07.351045 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.351001    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 23:11:07.352381 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.352369    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 23:11:07.352381 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.352383    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 23:11:07.352483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.352390    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 23:11:07.352483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.352395    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 23:11:07.352483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.352403    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 23:11:07.352483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.352408    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 23:11:07.352483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.352420    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 23:11:07.352483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.352426    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 23:11:07.352483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.352431    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 23:11:07.352483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.352437    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 23:11:07.352483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.352445    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 23:11:07.352483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.352450    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 23:11:07.353396 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.353387    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 23:11:07.353396 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.353397    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 23:11:07.354455 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.354414    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 23:11:07.354647 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.354600    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-184.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 23:11:07.358133 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.358114    2580 server.go:1268] "Started kubelet"
Sep 17 23:11:07.358199 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.358179    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 23:11:07.358308 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.358256    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 23:11:07.358394 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.358324    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 23:11:07.358823 ip-10-0-128-184 systemd[1]: Started Kubernetes Kubelet.
Sep 17 23:11:07.359094 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.359077    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-184.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 23:11:07.359964 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.359945    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 23:11:07.360369 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.360355    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 23:11:07.362336 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.362317    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 23:11:07.369408 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.369391    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 23:11:07.369486 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.369411    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 23:11:07.369534 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.369511    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:07.369603 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.368386    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-184.ec2.internal.18d63e8ab3217b32  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-184.ec2.internal,UID:ip-10-0-128-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-184.ec2.internal,},FirstTimestamp:2026-09-17 23:11:07.358075698 +0000 UTC m=+0.402451848,LastTimestamp:2026-09-17 23:11:07.358075698 +0000 UTC m=+0.402451848,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-184.ec2.internal,}"
Sep 17 23:11:07.370402 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.370384    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 23:11:07.370402 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.370402    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 23:11:07.370518 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.370400    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 23:11:07.370518 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.370499    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 23:11:07.370518 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.370505    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 23:11:07.370646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.370604    2580 factory.go:55] Registering systemd factory
Sep 17 23:11:07.370701 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.370684    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 17 23:11:07.370957 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.370940    2580 factory.go:153] Registering CRI-O factory
Sep 17 23:11:07.370957 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.370959    2580 factory.go:223] Registration of the crio container factory successfully
Sep 17 23:11:07.371070 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.371010    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 23:11:07.371070 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.371030    2580 factory.go:103] Registering Raw factory
Sep 17 23:11:07.371070 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.371043    2580 manager.go:1196] Started watching for new ooms in manager
Sep 17 23:11:07.371404 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.371390    2580 manager.go:349] Starting recovery of all containers
Sep 17 23:11:07.378972 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.378939    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-184.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 23:11:07.379274 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.379146    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 23:11:07.380902 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.380879    2580 manager.go:354] Recovery completed
Sep 17 23:11:07.384234 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.384218    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4mv4b"
Sep 17 23:11:07.385713 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.385701    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:07.387236 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.387223    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:07.387280 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.387250    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:07.387280 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.387264    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:07.387758 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.387746    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 23:11:07.387758 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.387757    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 23:11:07.387838 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.387770    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:11:07.387838 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.387757    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:07.388418 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.388406    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:07.388450 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.388429    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:07.388450 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.388438    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:07.388797 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.388729    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-184.ec2.internal.18d63e8ab4de6e2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-184.ec2.internal,UID:ip-10-0-128-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-184.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-184.ec2.internal,},FirstTimestamp:2026-09-17 23:11:07.387235885 +0000 UTC m=+0.431612035,LastTimestamp:2026-09-17 23:11:07.387235885 +0000 UTC m=+0.431612035,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-184.ec2.internal,}"
Sep 17 23:11:07.389973 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.389959    2580 policy_none.go:49] "None policy: Start"
Sep 17 23:11:07.389973 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.389966    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4mv4b"
Sep 17 23:11:07.389973 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.389974    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 23:11:07.390116 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.389985    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 23:11:07.393874 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.393861    2580 policy_none.go:47] "Start"
Sep 17 23:11:07.434363 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.434230    2580 manager.go:337] "Starting Device Plugin manager"
Sep 17 23:11:07.448968 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.434387    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 23:11:07.448968 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.434398    2580 server.go:87] "Starting device plugin registration server"
Sep 17 23:11:07.448968 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.434579    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 23:11:07.448968 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.434588    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 23:11:07.448968 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.434678    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 23:11:07.448968 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.434740    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 23:11:07.448968 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.434747    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 23:11:07.448968 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.435300    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 23:11:07.448968 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.435345    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:07.494093 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.494069    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 23:11:07.495337 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.495319    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 23:11:07.495409 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.495360    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 23:11:07.495409 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.495379    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 23:11:07.495495 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.495409    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 23:11:07.498339 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.498321    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:11:07.535640 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.535570    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:07.536404 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.536388    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:07.536462 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.536417    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:07.536462 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.536428    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:07.536462 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.536450    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.547502 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.547476    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.547502 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.547498    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-184.ec2.internal\": node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:07.588936 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.588915    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:07.595818 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.595800    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"]
Sep 17 23:11:07.595863 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.595856    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:07.597249 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.597236    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:07.597294 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.597262    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:07.597294 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.597272    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:07.599664 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.599653    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:07.599720 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.599705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.599756 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.599735    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:07.600347 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.600328    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:07.600431 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.600354    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:07.600431 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.600367    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:07.600431 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.600374    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:07.600431 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.600392    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:07.600431 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.600401    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:07.602580 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.602566    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.602648 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.602592    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:07.603211 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.603198    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:07.603282 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.603219    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:07.603282 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.603231    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:07.625154 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.625135    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-184.ec2.internal\" not found" node="ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.629312 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.629295    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-184.ec2.internal\" not found" node="ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.672601 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.672570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55afb46694881036d0e6730915a2cb2a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal\" (UID: \"55afb46694881036d0e6730915a2cb2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.672601 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.672598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b0094ac69bc5cb13f6f1d4e50931ad3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-184.ec2.internal\" (UID: \"3b0094ac69bc5cb13f6f1d4e50931ad3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.672799 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.672630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55afb46694881036d0e6730915a2cb2a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal\" (UID: \"55afb46694881036d0e6730915a2cb2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.689417 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.689396    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:07.773482 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.773444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55afb46694881036d0e6730915a2cb2a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal\" (UID: \"55afb46694881036d0e6730915a2cb2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.773558 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.773495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55afb46694881036d0e6730915a2cb2a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal\" (UID: \"55afb46694881036d0e6730915a2cb2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.773558 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.773512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b0094ac69bc5cb13f6f1d4e50931ad3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-184.ec2.internal\" (UID: \"3b0094ac69bc5cb13f6f1d4e50931ad3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.773558 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.773534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55afb46694881036d0e6730915a2cb2a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal\" (UID: \"55afb46694881036d0e6730915a2cb2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.773558 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.773547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b0094ac69bc5cb13f6f1d4e50931ad3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-184.ec2.internal\" (UID: \"3b0094ac69bc5cb13f6f1d4e50931ad3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.773722 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.773583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55afb46694881036d0e6730915a2cb2a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal\" (UID: \"55afb46694881036d0e6730915a2cb2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.789533 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.789489    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:07.890365 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.890328    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:07.928507 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.928484    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.932851 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:07.932829    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:07.991258 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:07.991218    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:08.091797 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:08.091721    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:08.192297 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:08.192260    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:08.292740 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:08.292706    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:08.303873 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:08.303854    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 23:11:08.303990 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:08.303975    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:11:08.393780 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:08.393726    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:08.395828 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:08.395806    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:06:08 +0000 UTC" deadline="2028-02-11 07:15:57.01911895 +0000 UTC"
Sep 17 23:11:08.395877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:08.395829    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12272h4m48.62329287s"
Sep 17 23:11:08.494515 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:08.494486    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:08.572250 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:08.572218    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:11:08.595719 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:08.595683    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:08.605253 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:08.605218    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55afb46694881036d0e6730915a2cb2a.slice/crio-e54b36d42a12337b8f8202c88beb96a6d093264cc3195ddddead4b8a6a0d17a0 WatchSource:0}: Error finding container e54b36d42a12337b8f8202c88beb96a6d093264cc3195ddddead4b8a6a0d17a0: Status 404 returned error can't find the container with id e54b36d42a12337b8f8202c88beb96a6d093264cc3195ddddead4b8a6a0d17a0
Sep 17 23:11:08.605779 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:08.605757    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0094ac69bc5cb13f6f1d4e50931ad3.slice/crio-f3fc2d92f493dbfd04445b3e571185da4c9044460b960e5e60a02c90b4b908da WatchSource:0}: Error finding container f3fc2d92f493dbfd04445b3e571185da4c9044460b960e5e60a02c90b4b908da: Status 404 returned error can't find the container with id f3fc2d92f493dbfd04445b3e571185da4c9044460b960e5e60a02c90b4b908da
Sep 17 23:11:08.610355 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:08.610342    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:11:08.695814 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:08.695746    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-184.ec2.internal\" not found"
Sep 17 23:11:08.699374 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:08.699358    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:11:08.756443 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:08.756418    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:11:08.770336 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:08.770317    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:08.786478 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:08.786455    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:11:08.787358 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:08.787346    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal"
Sep 17 23:11:08.796115 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:08.796101    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:11:09.208284 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.208258    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:11:09.346544 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.346511    2580 apiserver.go:52] "Watching apiserver"
Sep 17 23:11:09.354148 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.354114    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 23:11:09.354609 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.354584    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal","openshift-multus/multus-additional-cni-plugins-2p4f8","openshift-multus/multus-zgnlk","openshift-multus/network-metrics-daemon-gr7p6","openshift-network-operator/iptables-alerter-zn8l4","kube-system/global-pull-secret-syncer-xt5fb","kube-system/konnectivity-agent-pgkdk","kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5","openshift-dns/node-resolver-t65zf","openshift-network-diagnostics/network-check-target-8gs2j","openshift-ovn-kubernetes/ovnkube-node-jnsbp","openshift-cluster-node-tuning-operator/tuned-xdzcp","openshift-image-registry/node-ca-kcvpd"]
Sep 17 23:11:09.357327 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.357296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pgkdk"
Sep 17 23:11:09.359620 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.359600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.360239 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.360219    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 23:11:09.360471 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.360459    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 23:11:09.360644 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.360629    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hxqrc\""
Sep 17 23:11:09.361918 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.361894    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:11:09.362012 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.361952    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.362260 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.362242    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 23:11:09.362608 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.362586    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dfx24\""
Sep 17 23:11:09.364206 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.364180    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 23:11:09.364503 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.364484    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 23:11:09.365672 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.364747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-b42bs\""
Sep 17 23:11:09.365672 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.364966    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 23:11:09.365672 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.365128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 23:11:09.365672 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.365411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:09.365672 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.365495    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:09.367099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.367083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zn8l4"
Sep 17 23:11:09.370993 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.369240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:09.370993 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.369327    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:09.370993 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.369559    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:11:09.370993 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.369594    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ln6jh\""
Sep 17 23:11:09.370993 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.369621    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 23:11:09.370993 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.369877    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 23:11:09.370993 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.370054    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 23:11:09.371830 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.371810    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.374098 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.374076    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 23:11:09.374316 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.374300    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 23:11:09.374666 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.374616    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 23:11:09.374755 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.374701    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hmcg4\""
Sep 17 23:11:09.374818 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.374757    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kcvpd"
Sep 17 23:11:09.377788 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.376995    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vspd5\""
Sep 17 23:11:09.377788 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.377012    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 23:11:09.377788 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.377053    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 23:11:09.377788 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.377337    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 23:11:09.377788 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.377632    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.382562 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.382250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t65zf"
Sep 17 23:11:09.383138 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.383102    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 23:11:09.383138 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.383133    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 23:11:09.383319 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.383200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-socket-dir\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.383319 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.383242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdnxv\" (UniqueName: \"kubernetes.io/projected/138a1459-9d12-4a9a-8d9d-b34189a27764-kube-api-access-xdnxv\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.383319 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.383274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:09.383484 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.383326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482cg\" (UniqueName: \"kubernetes.io/projected/1e6da33b-9d1e-472a-97aa-e4951d3972ff-kube-api-access-482cg\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:09.383484 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.383387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-sys-fs\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.383952 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.383545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-cnibin\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.383952 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.383592    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7wc8s\""
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-cni-binary-copy\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-multus-conf-dir\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-run-multus-certs\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-etc-kubernetes\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39469e70-5457-494d-9f87-7cf3f8c7d557-iptables-alerter-script\") pod \"iptables-alerter-zn8l4\" (UID: \"39469e70-5457-494d-9f87-7cf3f8c7d557\") " pod="openshift-network-operator/iptables-alerter-zn8l4"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70431036-0857-4c0f-83f5-842c5b84a114-agent-certs\") pod \"konnectivity-agent-pgkdk\" (UID: \"70431036-0857-4c0f-83f5-842c5b84a114\") " pod="kube-system/konnectivity-agent-pgkdk"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-multus-cni-dir\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-multus-socket-dir-parent\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384687    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-run-netns\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n7lv\" (UniqueName: \"kubernetes.io/projected/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-kube-api-access-6n7lv\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-run\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-sys\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70431036-0857-4c0f-83f5-842c5b84a114-konnectivity-ca\") pod \"konnectivity-agent-pgkdk\" (UID: \"70431036-0857-4c0f-83f5-842c5b84a114\") " pod="kube-system/konnectivity-agent-pgkdk"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-system-cni-dir\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384852    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v6jtc\""
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-multus-daemon-config\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.384888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-lib-modules\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-host\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.384988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39469e70-5457-494d-9f87-7cf3f8c7d557-host-slash\") pod \"iptables-alerter-zn8l4\" (UID: \"39469e70-5457-494d-9f87-7cf3f8c7d557\") " pod="openshift-network-operator/iptables-alerter-zn8l4"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385021    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-registration-dir\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-kubelet-config\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-dbus\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-sysconfig\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-var-lib-kubelet\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-etc-selinux\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-modprobe-d\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.385307    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-kubernetes\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-tuned\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.385791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-device-dir\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.386545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtmj\" (UniqueName: \"kubernetes.io/projected/3d06769a-458c-4280-805b-c24da91a066a-kube-api-access-nvtmj\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.386545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-var-lib-cni-multus\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.386545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.385883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-hostroot\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.386545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.386038    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 23:11:09.386545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.386026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-sysctl-d\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.386545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.386086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-sysctl-conf\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.386545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.386121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-systemd\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.386545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.386156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-run-k8s-cni-cncf-io\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.386545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.386188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-var-lib-cni-bin\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.386545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.386473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-var-lib-kubelet\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.386545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.386505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/138a1459-9d12-4a9a-8d9d-b34189a27764-tmp\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.386545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.386542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6kk\" (UniqueName: \"kubernetes.io/projected/39469e70-5457-494d-9f87-7cf3f8c7d557-kube-api-access-cg6kk\") pod \"iptables-alerter-zn8l4\" (UID: \"39469e70-5457-494d-9f87-7cf3f8c7d557\") " pod="openshift-network-operator/iptables-alerter-zn8l4"
Sep 17 23:11:09.387212 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.386987    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:11:09.387842 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.387821    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.390105 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.390080    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 23:11:09.390380 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.390359    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 23:11:09.390449 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.390396    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 23:11:09.390507 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.390471    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 23:11:09.391496 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.391475    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nqsjd\""
Sep 17 23:11:09.391583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.391564    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 23:11:09.391725 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.391707    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 23:11:09.409866 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.409848    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zjqg7"
Sep 17 23:11:09.418637 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.418607    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zjqg7"
Sep 17 23:11:09.471453 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.471387    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 23:11:09.486710 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.486686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdnxv\" (UniqueName: \"kubernetes.io/projected/138a1459-9d12-4a9a-8d9d-b34189a27764-kube-api-access-xdnxv\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.486844 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.486715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:09.486844 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.486737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/405ad021-9316-4805-b71b-3a6a13854a1d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.486844 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.486754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-etc-openvswitch\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.486844 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.486778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-cnibin\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.486844 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.486800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-etc-kubernetes\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.486844 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.486823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39469e70-5457-494d-9f87-7cf3f8c7d557-iptables-alerter-script\") pod \"iptables-alerter-zn8l4\" (UID: \"39469e70-5457-494d-9f87-7cf3f8c7d557\") " pod="openshift-network-operator/iptables-alerter-zn8l4"
Sep 17 23:11:09.487078 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.486847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/405ad021-9316-4805-b71b-3a6a13854a1d-system-cni-dir\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.487078 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.486862    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:09.487078 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.486873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-kubelet\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.487078 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.486918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-systemd-units\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.487078 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.486942    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs podName:1e6da33b-9d1e-472a-97aa-e4951d3972ff nodeName:}" failed. No retries permitted until 2026-09-17 23:11:09.986919809 +0000 UTC m=+3.031295968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs") pod "network-metrics-daemon-gr7p6" (UID: "1e6da33b-9d1e-472a-97aa-e4951d3972ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:09.487078 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-cnibin\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-etc-kubernetes\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-multus-cni-dir\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-multus-socket-dir-parent\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n7lv\" (UniqueName: \"kubernetes.io/projected/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-kube-api-access-6n7lv\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-run\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-sys\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-multus-socket-dir-parent\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-lib-modules\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-multus-cni-dir\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-run\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-sys\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-run-ovn-kubernetes\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvzz6\" (UniqueName: \"kubernetes.io/projected/bbd99a55-d6c1-44c0-adda-255aab5fb532-kube-api-access-gvzz6\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-lib-modules\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.487405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-system-cni-dir\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39469e70-5457-494d-9f87-7cf3f8c7d557-host-slash\") pod \"iptables-alerter-zn8l4\" (UID: \"39469e70-5457-494d-9f87-7cf3f8c7d557\") " pod="openshift-network-operator/iptables-alerter-zn8l4"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-etc-selinux\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5-hosts-file\") pod \"node-resolver-t65zf\" (UID: \"60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5\") " pod="openshift-dns/node-resolver-t65zf"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8dz\" (UniqueName: \"kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz\") pod \"network-check-target-8gs2j\" (UID: \"50270559-ac36-4cd1-85e4-1267baf1b3cc\") " pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39469e70-5457-494d-9f87-7cf3f8c7d557-host-slash\") pod \"iptables-alerter-zn8l4\" (UID: \"39469e70-5457-494d-9f87-7cf3f8c7d557\") " pod="openshift-network-operator/iptables-alerter-zn8l4"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-run-openvswitch\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-system-cni-dir\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39469e70-5457-494d-9f87-7cf3f8c7d557-iptables-alerter-script\") pod \"iptables-alerter-zn8l4\" (UID: \"39469e70-5457-494d-9f87-7cf3f8c7d557\") " pod="openshift-network-operator/iptables-alerter-zn8l4"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bbd99a55-d6c1-44c0-adda-255aab5fb532-ovnkube-script-lib\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487603    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-etc-selinux\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-kubelet-config\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjfmx\" (UniqueName: \"kubernetes.io/projected/265e74c8-a31a-43de-af18-711758cc8e5d-kube-api-access-tjfmx\") pod \"node-ca-kcvpd\" (UID: \"265e74c8-a31a-43de-af18-711758cc8e5d\") " pod="openshift-image-registry/node-ca-kcvpd"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/405ad021-9316-4805-b71b-3a6a13854a1d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5-tmp-dir\") pod \"node-resolver-t65zf\" (UID: \"60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5\") " pod="openshift-dns/node-resolver-t65zf"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-modprobe-d\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.488099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-kubelet-config\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bbd99a55-d6c1-44c0-adda-255aab5fb532-env-overrides\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-hostroot\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-sysctl-conf\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-modprobe-d\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-run-ovn\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-hostroot\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-run-k8s-cni-cncf-io\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-var-lib-cni-bin\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-sysctl-conf\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-var-lib-kubelet\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.487986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-run-k8s-cni-cncf-io\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/138a1459-9d12-4a9a-8d9d-b34189a27764-tmp\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-var-lib-cni-bin\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-socket-dir\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c757b\" (UniqueName: \"kubernetes.io/projected/60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5-kube-api-access-c757b\") pod \"node-resolver-t65zf\" (UID: \"60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5\") " pod="openshift-dns/node-resolver-t65zf"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-slash\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.489092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-var-lib-kubelet\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bbd99a55-d6c1-44c0-adda-255aab5fb532-ovnkube-config\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482cg\" (UniqueName: \"kubernetes.io/projected/1e6da33b-9d1e-472a-97aa-e4951d3972ff-kube-api-access-482cg\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-sys-fs\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46kll\" (UniqueName: \"kubernetes.io/projected/405ad021-9316-4805-b71b-3a6a13854a1d-kube-api-access-46kll\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-run-systemd\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bbd99a55-d6c1-44c0-adda-255aab5fb532-ovn-node-metrics-cert\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-cni-binary-copy\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-sys-fs\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-socket-dir\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-multus-conf-dir\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-run-multus-certs\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488362    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/265e74c8-a31a-43de-af18-711758cc8e5d-host\") pod \"node-ca-kcvpd\" (UID: \"265e74c8-a31a-43de-af18-711758cc8e5d\") " pod="openshift-image-registry/node-ca-kcvpd"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-multus-conf-dir\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-cni-bin\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70431036-0857-4c0f-83f5-842c5b84a114-agent-certs\") pod \"konnectivity-agent-pgkdk\" (UID: \"70431036-0857-4c0f-83f5-842c5b84a114\") " pod="kube-system/konnectivity-agent-pgkdk"
Sep 17 23:11:09.490071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-run-netns\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-host\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-registration-dir\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtmj\" (UniqueName: \"kubernetes.io/projected/3d06769a-458c-4280-805b-c24da91a066a-kube-api-access-nvtmj\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-run-netns\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/265e74c8-a31a-43de-af18-711758cc8e5d-serviceca\") pod \"node-ca-kcvpd\" (UID: \"265e74c8-a31a-43de-af18-711758cc8e5d\") " pod="openshift-image-registry/node-ca-kcvpd"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-var-lib-openvswitch\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70431036-0857-4c0f-83f5-842c5b84a114-konnectivity-ca\") pod \"konnectivity-agent-pgkdk\" (UID: \"70431036-0857-4c0f-83f5-842c5b84a114\") " pod="kube-system/konnectivity-agent-pgkdk"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-multus-daemon-config\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-node-log\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-log-socket\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-cni-netd\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-dbus\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-sysconfig\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-var-lib-kubelet\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-run-multus-certs\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.491165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-run-netns\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.488980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-kubernetes\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-tuned\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-dbus\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-device-dir\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489084    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-registration-dir\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/405ad021-9316-4805-b71b-3a6a13854a1d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-sysconfig\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-var-lib-cni-multus\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-sysctl-d\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-var-lib-kubelet\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-systemd\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d06769a-458c-4280-805b-c24da91a066a-device-dir\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/405ad021-9316-4805-b71b-3a6a13854a1d-cni-binary-copy\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6kk\" (UniqueName: \"kubernetes.io/projected/39469e70-5457-494d-9f87-7cf3f8c7d557-kube-api-access-cg6kk\") pod \"iptables-alerter-zn8l4\" (UID: \"39469e70-5457-494d-9f87-7cf3f8c7d557\") " pod="openshift-network-operator/iptables-alerter-zn8l4"
Sep 17 23:11:09.492018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/405ad021-9316-4805-b71b-3a6a13854a1d-cnibin\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.492726 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-cni-binary-copy\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.492726 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.489360    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:09.492726 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.489399    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret podName:0bc41eaf-0a42-4187-8eec-c27fd3ab26d8 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:09.989385534 +0000 UTC m=+3.033761673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret") pod "global-pull-secret-syncer-xt5fb" (UID: "0bc41eaf-0a42-4187-8eec-c27fd3ab26d8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:09.492726 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-systemd\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.492726 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489606    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70431036-0857-4c0f-83f5-842c5b84a114-konnectivity-ca\") pod \"konnectivity-agent-pgkdk\" (UID: \"70431036-0857-4c0f-83f5-842c5b84a114\") " pod="kube-system/konnectivity-agent-pgkdk"
Sep 17 23:11:09.492726 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489636    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-host\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.492726 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-kubernetes\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.492726 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-sysctl-d\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.492726 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-host-var-lib-cni-multus\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.492726 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.489832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-multus-daemon-config\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.492726 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.492554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/138a1459-9d12-4a9a-8d9d-b34189a27764-etc-tuned\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.493146 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.492834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70431036-0857-4c0f-83f5-842c5b84a114-agent-certs\") pod \"konnectivity-agent-pgkdk\" (UID: \"70431036-0857-4c0f-83f5-842c5b84a114\") " pod="kube-system/konnectivity-agent-pgkdk"
Sep 17 23:11:09.493518 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.493499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/138a1459-9d12-4a9a-8d9d-b34189a27764-tmp\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.498142 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.498115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdnxv\" (UniqueName: \"kubernetes.io/projected/138a1459-9d12-4a9a-8d9d-b34189a27764-kube-api-access-xdnxv\") pod \"tuned-xdzcp\" (UID: \"138a1459-9d12-4a9a-8d9d-b34189a27764\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.499576 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.499533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n7lv\" (UniqueName: \"kubernetes.io/projected/ff87e86b-3f0f-4ab5-ba79-86dae21e2c95-kube-api-access-6n7lv\") pod \"multus-zgnlk\" (UID: \"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95\") " pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.501515 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.501491    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-482cg\" (UniqueName: \"kubernetes.io/projected/1e6da33b-9d1e-472a-97aa-e4951d3972ff-kube-api-access-482cg\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:09.502024 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.501928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6kk\" (UniqueName: \"kubernetes.io/projected/39469e70-5457-494d-9f87-7cf3f8c7d557-kube-api-access-cg6kk\") pod \"iptables-alerter-zn8l4\" (UID: \"39469e70-5457-494d-9f87-7cf3f8c7d557\") " pod="openshift-network-operator/iptables-alerter-zn8l4"
Sep 17 23:11:09.502643 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.502612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtmj\" (UniqueName: \"kubernetes.io/projected/3d06769a-458c-4280-805b-c24da91a066a-kube-api-access-nvtmj\") pod \"aws-ebs-csi-driver-node-tfvl5\" (UID: \"3d06769a-458c-4280-805b-c24da91a066a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.504452 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.504342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal" event={"ID":"3b0094ac69bc5cb13f6f1d4e50931ad3","Type":"ContainerStarted","Data":"f3fc2d92f493dbfd04445b3e571185da4c9044460b960e5e60a02c90b4b908da"}
Sep 17 23:11:09.505709 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.505662    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal" event={"ID":"55afb46694881036d0e6730915a2cb2a","Type":"ContainerStarted","Data":"e54b36d42a12337b8f8202c88beb96a6d093264cc3195ddddead4b8a6a0d17a0"}
Sep 17 23:11:09.589906 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.589875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/265e74c8-a31a-43de-af18-711758cc8e5d-serviceca\") pod \"node-ca-kcvpd\" (UID: \"265e74c8-a31a-43de-af18-711758cc8e5d\") " pod="openshift-image-registry/node-ca-kcvpd"
Sep 17 23:11:09.589906 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.589908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-var-lib-openvswitch\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.589934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-node-log\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.589952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-log-socket\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.589975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-cni-netd\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.589999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-var-lib-openvswitch\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-node-log\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-run-netns\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-run-netns\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-log-socket\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-cni-netd\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/405ad021-9316-4805-b71b-3a6a13854a1d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.590163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/405ad021-9316-4805-b71b-3a6a13854a1d-cni-binary-copy\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.590163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/405ad021-9316-4805-b71b-3a6a13854a1d-cnibin\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/405ad021-9316-4805-b71b-3a6a13854a1d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-etc-openvswitch\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/405ad021-9316-4805-b71b-3a6a13854a1d-system-cni-dir\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/405ad021-9316-4805-b71b-3a6a13854a1d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-kubelet\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/265e74c8-a31a-43de-af18-711758cc8e5d-serviceca\") pod \"node-ca-kcvpd\" (UID: \"265e74c8-a31a-43de-af18-711758cc8e5d\") " pod="openshift-image-registry/node-ca-kcvpd"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-kubelet\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-etc-openvswitch\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/405ad021-9316-4805-b71b-3a6a13854a1d-system-cni-dir\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/405ad021-9316-4805-b71b-3a6a13854a1d-cnibin\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-systemd-units\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-run-ovn-kubernetes\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvzz6\" (UniqueName: \"kubernetes.io/projected/bbd99a55-d6c1-44c0-adda-255aab5fb532-kube-api-access-gvzz6\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5-hosts-file\") pod \"node-resolver-t65zf\" (UID: \"60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5\") " pod="openshift-dns/node-resolver-t65zf"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8dz\" (UniqueName: \"kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz\") pod \"network-check-target-8gs2j\" (UID: \"50270559-ac36-4cd1-85e4-1267baf1b3cc\") " pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-run-openvswitch\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.590789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bbd99a55-d6c1-44c0-adda-255aab5fb532-ovnkube-script-lib\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjfmx\" (UniqueName: \"kubernetes.io/projected/265e74c8-a31a-43de-af18-711758cc8e5d-kube-api-access-tjfmx\") pod \"node-ca-kcvpd\" (UID: \"265e74c8-a31a-43de-af18-711758cc8e5d\") " pod="openshift-image-registry/node-ca-kcvpd"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/405ad021-9316-4805-b71b-3a6a13854a1d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5-tmp-dir\") pod \"node-resolver-t65zf\" (UID: \"60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5\") " pod="openshift-dns/node-resolver-t65zf"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bbd99a55-d6c1-44c0-adda-255aab5fb532-env-overrides\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-run-ovn\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/405ad021-9316-4805-b71b-3a6a13854a1d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-run-ovn-kubernetes\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5-hosts-file\") pod \"node-resolver-t65zf\" (UID: \"60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5\") " pod="openshift-dns/node-resolver-t65zf"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/405ad021-9316-4805-b71b-3a6a13854a1d-cni-binary-copy\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c757b\" (UniqueName: \"kubernetes.io/projected/60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5-kube-api-access-c757b\") pod \"node-resolver-t65zf\" (UID: \"60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5\") " pod="openshift-dns/node-resolver-t65zf"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-run-openvswitch\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-slash\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bbd99a55-d6c1-44c0-adda-255aab5fb532-ovnkube-config\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46kll\" (UniqueName: \"kubernetes.io/projected/405ad021-9316-4805-b71b-3a6a13854a1d-kube-api-access-46kll\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.590998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-run-systemd\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.591617 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bbd99a55-d6c1-44c0-adda-255aab5fb532-ovn-node-metrics-cert\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/265e74c8-a31a-43de-af18-711758cc8e5d-host\") pod \"node-ca-kcvpd\" (UID: \"265e74c8-a31a-43de-af18-711758cc8e5d\") " pod="openshift-image-registry/node-ca-kcvpd"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-cni-bin\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-cni-bin\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-run-systemd\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-systemd-units\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-slash\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bbd99a55-d6c1-44c0-adda-255aab5fb532-ovnkube-script-lib\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591528    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bbd99a55-d6c1-44c0-adda-255aab5fb532-ovnkube-config\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bbd99a55-d6c1-44c0-adda-255aab5fb532-run-ovn\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bbd99a55-d6c1-44c0-adda-255aab5fb532-env-overrides\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.591980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/265e74c8-a31a-43de-af18-711758cc8e5d-host\") pod \"node-ca-kcvpd\" (UID: \"265e74c8-a31a-43de-af18-711758cc8e5d\") " pod="openshift-image-registry/node-ca-kcvpd"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.592020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/405ad021-9316-4805-b71b-3a6a13854a1d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.592438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.592337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5-tmp-dir\") pod \"node-resolver-t65zf\" (UID: \"60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5\") " pod="openshift-dns/node-resolver-t65zf"
Sep 17 23:11:09.593772 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.593753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bbd99a55-d6c1-44c0-adda-255aab5fb532-ovn-node-metrics-cert\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.598804 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.598783    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:09.598909 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.598809    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:09.598909 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.598824    2580 projected.go:196] Error preparing data for projected volume kube-api-access-6c8dz for pod openshift-network-diagnostics/network-check-target-8gs2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:09.598909 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.598883    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz podName:50270559-ac36-4cd1-85e4-1267baf1b3cc nodeName:}" failed. No retries permitted until 2026-09-17 23:11:10.098863936 +0000 UTC m=+3.143240081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6c8dz" (UniqueName: "kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz") pod "network-check-target-8gs2j" (UID: "50270559-ac36-4cd1-85e4-1267baf1b3cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:09.601427 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.601403    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46kll\" (UniqueName: \"kubernetes.io/projected/405ad021-9316-4805-b71b-3a6a13854a1d-kube-api-access-46kll\") pod \"multus-additional-cni-plugins-2p4f8\" (UID: \"405ad021-9316-4805-b71b-3a6a13854a1d\") " pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.601527 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.601468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjfmx\" (UniqueName: \"kubernetes.io/projected/265e74c8-a31a-43de-af18-711758cc8e5d-kube-api-access-tjfmx\") pod \"node-ca-kcvpd\" (UID: \"265e74c8-a31a-43de-af18-711758cc8e5d\") " pod="openshift-image-registry/node-ca-kcvpd"
Sep 17 23:11:09.601963 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.601940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c757b\" (UniqueName: \"kubernetes.io/projected/60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5-kube-api-access-c757b\") pod \"node-resolver-t65zf\" (UID: \"60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5\") " pod="openshift-dns/node-resolver-t65zf"
Sep 17 23:11:09.603338 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.603316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvzz6\" (UniqueName: \"kubernetes.io/projected/bbd99a55-d6c1-44c0-adda-255aab5fb532-kube-api-access-gvzz6\") pod \"ovnkube-node-jnsbp\" (UID: \"bbd99a55-d6c1-44c0-adda-255aab5fb532\") " pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.671781 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.671747    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pgkdk"
Sep 17 23:11:09.681139 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.681117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xdzcp"
Sep 17 23:11:09.692090 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.692060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zgnlk"
Sep 17 23:11:09.700894 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.700874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zn8l4"
Sep 17 23:11:09.709831 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.709812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5"
Sep 17 23:11:09.716462 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.716427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kcvpd"
Sep 17 23:11:09.732355 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.732306    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2p4f8"
Sep 17 23:11:09.740044 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.740016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t65zf"
Sep 17 23:11:09.745804 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.745786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:09.994950 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.994866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:09.994950 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:09.994922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:09.995154 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.995022    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:09.995154 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.995038    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:09.995154 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.995097    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs podName:1e6da33b-9d1e-472a-97aa-e4951d3972ff nodeName:}" failed. No retries permitted until 2026-09-17 23:11:10.995078751 +0000 UTC m=+4.039454893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs") pod "network-metrics-daemon-gr7p6" (UID: "1e6da33b-9d1e-472a-97aa-e4951d3972ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:09.995154 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:09.995118    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret podName:0bc41eaf-0a42-4187-8eec-c27fd3ab26d8 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:10.995109309 +0000 UTC m=+4.039485448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret") pod "global-pull-secret-syncer-xt5fb" (UID: "0bc41eaf-0a42-4187-8eec-c27fd3ab26d8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:10.197242 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:10.197202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8dz\" (UniqueName: \"kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz\") pod \"network-check-target-8gs2j\" (UID: \"50270559-ac36-4cd1-85e4-1267baf1b3cc\") " pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:10.197406 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:10.197366    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:10.197406 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:10.197390    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:10.197406 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:10.197402    2580 projected.go:196] Error preparing data for projected volume kube-api-access-6c8dz for pod openshift-network-diagnostics/network-check-target-8gs2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:10.197575 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:10.197490    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz podName:50270559-ac36-4cd1-85e4-1267baf1b3cc nodeName:}" failed. No retries permitted until 2026-09-17 23:11:11.197470252 +0000 UTC m=+4.241846406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6c8dz" (UniqueName: "kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz") pod "network-check-target-8gs2j" (UID: "50270559-ac36-4cd1-85e4-1267baf1b3cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:10.421740 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:10.421701    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:06:10 +0000 UTC" deadline="2028-03-28 21:04:32.733726624 +0000 UTC"
Sep 17 23:11:10.421740 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:10.421732    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13389h53m22.311998485s"
Sep 17 23:11:10.572414 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:10.572379    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bb6bc0_7d80_4b8a_8d28_b27636b5e7a5.slice/crio-2e5563d1bd079191b5f092e445b8c12ef70a50ce83c873d927c464b57048ebda WatchSource:0}: Error finding container 2e5563d1bd079191b5f092e445b8c12ef70a50ce83c873d927c464b57048ebda: Status 404 returned error can't find the container with id 2e5563d1bd079191b5f092e445b8c12ef70a50ce83c873d927c464b57048ebda
Sep 17 23:11:10.579052 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:10.579019    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff87e86b_3f0f_4ab5_ba79_86dae21e2c95.slice/crio-a8c875696505e1409531b535e6b4b1305a96f2f27dd890a3b64fa4312a4d27be WatchSource:0}: Error finding container a8c875696505e1409531b535e6b4b1305a96f2f27dd890a3b64fa4312a4d27be: Status 404 returned error can't find the container with id a8c875696505e1409531b535e6b4b1305a96f2f27dd890a3b64fa4312a4d27be
Sep 17 23:11:10.585950 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:10.585915    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd99a55_d6c1_44c0_adda_255aab5fb532.slice/crio-177e482b3fda0a217f991ee759659c91d0b6347f3c6237d1f882aeb88fb5056a WatchSource:0}: Error finding container 177e482b3fda0a217f991ee759659c91d0b6347f3c6237d1f882aeb88fb5056a: Status 404 returned error can't find the container with id 177e482b3fda0a217f991ee759659c91d0b6347f3c6237d1f882aeb88fb5056a
Sep 17 23:11:10.586781 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:10.586761    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d06769a_458c_4280_805b_c24da91a066a.slice/crio-da5515ce5b0f4d8af94ae7f0175a569bb7539f8d6b7b56f6a60237700be22153 WatchSource:0}: Error finding container da5515ce5b0f4d8af94ae7f0175a569bb7539f8d6b7b56f6a60237700be22153: Status 404 returned error can't find the container with id da5515ce5b0f4d8af94ae7f0175a569bb7539f8d6b7b56f6a60237700be22153
Sep 17 23:11:10.587682 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:10.587617    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138a1459_9d12_4a9a_8d9d_b34189a27764.slice/crio-ff806ac04adf8d1e11e57d2c645471d22620917518454b43aeeab0670a77ee0d WatchSource:0}: Error finding container ff806ac04adf8d1e11e57d2c645471d22620917518454b43aeeab0670a77ee0d: Status 404 returned error can't find the container with id ff806ac04adf8d1e11e57d2c645471d22620917518454b43aeeab0670a77ee0d
Sep 17 23:11:10.588481 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:10.588431    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405ad021_9316_4805_b71b_3a6a13854a1d.slice/crio-19b9b11f4c779c3e0394ffef2aec1510ce37ffbf27df9518ed8ccb63fa5e18e0 WatchSource:0}: Error finding container 19b9b11f4c779c3e0394ffef2aec1510ce37ffbf27df9518ed8ccb63fa5e18e0: Status 404 returned error can't find the container with id 19b9b11f4c779c3e0394ffef2aec1510ce37ffbf27df9518ed8ccb63fa5e18e0
Sep 17 23:11:10.589600 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:10.589500    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70431036_0857_4c0f_83f5_842c5b84a114.slice/crio-2667e354f5db8ed14f363551fa8e56f1d51aa342df998ffa50312f1c997e4662 WatchSource:0}: Error finding container 2667e354f5db8ed14f363551fa8e56f1d51aa342df998ffa50312f1c997e4662: Status 404 returned error can't find the container with id 2667e354f5db8ed14f363551fa8e56f1d51aa342df998ffa50312f1c997e4662
Sep 17 23:11:10.590535 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:10.590470    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39469e70_5457_494d_9f87_7cf3f8c7d557.slice/crio-d842674eb2ffe3d5e21af6156d78968199d20beb362ab7e72fd9aafb7febbeaa WatchSource:0}: Error finding container d842674eb2ffe3d5e21af6156d78968199d20beb362ab7e72fd9aafb7febbeaa: Status 404 returned error can't find the container with id d842674eb2ffe3d5e21af6156d78968199d20beb362ab7e72fd9aafb7febbeaa
Sep 17 23:11:10.592444 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:10.592255    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265e74c8_a31a_43de_af18_711758cc8e5d.slice/crio-ce9eac41571f1719602e97b83a571f846402af32a6a91852086ab0a038e8df7f WatchSource:0}: Error finding container ce9eac41571f1719602e97b83a571f846402af32a6a91852086ab0a038e8df7f: Status 404 returned error can't find the container with id ce9eac41571f1719602e97b83a571f846402af32a6a91852086ab0a038e8df7f
Sep 17 23:11:11.002257 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.002030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:11.002396 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.002291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:11.002396 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:11.002177    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:11.002396 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:11.002391    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:11.002523 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:11.002413    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret podName:0bc41eaf-0a42-4187-8eec-c27fd3ab26d8 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:13.002392539 +0000 UTC m=+6.046768693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret") pod "global-pull-secret-syncer-xt5fb" (UID: "0bc41eaf-0a42-4187-8eec-c27fd3ab26d8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:11.002523 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:11.002432    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs podName:1e6da33b-9d1e-472a-97aa-e4951d3972ff nodeName:}" failed. No retries permitted until 2026-09-17 23:11:13.002420524 +0000 UTC m=+6.046796663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs") pod "network-metrics-daemon-gr7p6" (UID: "1e6da33b-9d1e-472a-97aa-e4951d3972ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:11.204078 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.203269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8dz\" (UniqueName: \"kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz\") pod \"network-check-target-8gs2j\" (UID: \"50270559-ac36-4cd1-85e4-1267baf1b3cc\") " pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:11.204078 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:11.203477    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:11.204078 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:11.203502    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:11.204078 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:11.203515    2580 projected.go:196] Error preparing data for projected volume kube-api-access-6c8dz for pod openshift-network-diagnostics/network-check-target-8gs2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:11.204078 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:11.203575    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz podName:50270559-ac36-4cd1-85e4-1267baf1b3cc nodeName:}" failed. No retries permitted until 2026-09-17 23:11:13.203556791 +0000 UTC m=+6.247932935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6c8dz" (UniqueName: "kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz") pod "network-check-target-8gs2j" (UID: "50270559-ac36-4cd1-85e4-1267baf1b3cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:11.422398 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.422333    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:06:10 +0000 UTC" deadline="2028-06-29 18:34:47.804166834 +0000 UTC"
Sep 17 23:11:11.422398 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.422396    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15619h23m36.381775469s"
Sep 17 23:11:11.498762 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.498671    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:11.498956 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:11.498825    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:11.498956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.498915    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:11.499076 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:11.499008    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:11.499076 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.499070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:11.499203 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:11.499142    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:11.538785 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.538746    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xdzcp" event={"ID":"138a1459-9d12-4a9a-8d9d-b34189a27764","Type":"ContainerStarted","Data":"ff806ac04adf8d1e11e57d2c645471d22620917518454b43aeeab0670a77ee0d"}
Sep 17 23:11:11.555607 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.555571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pgkdk" event={"ID":"70431036-0857-4c0f-83f5-842c5b84a114","Type":"ContainerStarted","Data":"2667e354f5db8ed14f363551fa8e56f1d51aa342df998ffa50312f1c997e4662"}
Sep 17 23:11:11.559008 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.558977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5" event={"ID":"3d06769a-458c-4280-805b-c24da91a066a","Type":"ContainerStarted","Data":"da5515ce5b0f4d8af94ae7f0175a569bb7539f8d6b7b56f6a60237700be22153"}
Sep 17 23:11:11.579009 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.578962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp" event={"ID":"bbd99a55-d6c1-44c0-adda-255aab5fb532","Type":"ContainerStarted","Data":"177e482b3fda0a217f991ee759659c91d0b6347f3c6237d1f882aeb88fb5056a"}
Sep 17 23:11:11.598726 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.595893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zgnlk" event={"ID":"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95","Type":"ContainerStarted","Data":"a8c875696505e1409531b535e6b4b1305a96f2f27dd890a3b64fa4312a4d27be"}
Sep 17 23:11:11.606050 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.604965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal" event={"ID":"3b0094ac69bc5cb13f6f1d4e50931ad3","Type":"ContainerStarted","Data":"5724dc83815e405bde5336514cbd5a2c657ba0a2a78cdd66d33c605e2e8216b2"}
Sep 17 23:11:11.616430 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.616374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kcvpd" event={"ID":"265e74c8-a31a-43de-af18-711758cc8e5d","Type":"ContainerStarted","Data":"ce9eac41571f1719602e97b83a571f846402af32a6a91852086ab0a038e8df7f"}
Sep 17 23:11:11.625126 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.624234    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-184.ec2.internal" podStartSLOduration=2.624213948 podStartE2EDuration="2.624213948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:11.62395962 +0000 UTC m=+4.668335789" watchObservedRunningTime="2026-09-17 23:11:11.624213948 +0000 UTC m=+4.668590111"
Sep 17 23:11:11.639340 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.639157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t65zf" event={"ID":"60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5","Type":"ContainerStarted","Data":"2e5563d1bd079191b5f092e445b8c12ef70a50ce83c873d927c464b57048ebda"}
Sep 17 23:11:11.652799 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.652772    2580 generic.go:356] "Generic (PLEG): container finished" podID="55afb46694881036d0e6730915a2cb2a" containerID="5d5c0b46d6eab9446d98abbd09db663513c606f0755b984bd52370f4bc0a93c4" exitCode=0
Sep 17 23:11:11.652912 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.652865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal" event={"ID":"55afb46694881036d0e6730915a2cb2a","Type":"ContainerDied","Data":"5d5c0b46d6eab9446d98abbd09db663513c606f0755b984bd52370f4bc0a93c4"}
Sep 17 23:11:11.670506 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.670334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zn8l4" event={"ID":"39469e70-5457-494d-9f87-7cf3f8c7d557","Type":"ContainerStarted","Data":"d842674eb2ffe3d5e21af6156d78968199d20beb362ab7e72fd9aafb7febbeaa"}
Sep 17 23:11:11.685689 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:11.685637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2p4f8" event={"ID":"405ad021-9316-4805-b71b-3a6a13854a1d","Type":"ContainerStarted","Data":"19b9b11f4c779c3e0394ffef2aec1510ce37ffbf27df9518ed8ccb63fa5e18e0"}
Sep 17 23:11:12.715859 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:12.715774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal" event={"ID":"55afb46694881036d0e6730915a2cb2a","Type":"ContainerStarted","Data":"c2deaec21c8b3ff760144ee2077c8247ccd6b13892955ebb1b6aeb22c2154404"}
Sep 17 23:11:13.020468 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:13.020387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:13.020468 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:13.020451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:13.020747 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:13.020718    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:13.020810 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:13.020790    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs podName:1e6da33b-9d1e-472a-97aa-e4951d3972ff nodeName:}" failed. No retries permitted until 2026-09-17 23:11:17.020769102 +0000 UTC m=+10.065145269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs") pod "network-metrics-daemon-gr7p6" (UID: "1e6da33b-9d1e-472a-97aa-e4951d3972ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:13.021224 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:13.021199    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:13.021339 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:13.021264    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret podName:0bc41eaf-0a42-4187-8eec-c27fd3ab26d8 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:17.021243583 +0000 UTC m=+10.065619726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret") pod "global-pull-secret-syncer-xt5fb" (UID: "0bc41eaf-0a42-4187-8eec-c27fd3ab26d8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:13.223067 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:13.223028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8dz\" (UniqueName: \"kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz\") pod \"network-check-target-8gs2j\" (UID: \"50270559-ac36-4cd1-85e4-1267baf1b3cc\") " pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:13.223258 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:13.223205    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:13.223258 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:13.223227    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:13.223258 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:13.223242    2580 projected.go:196] Error preparing data for projected volume kube-api-access-6c8dz for pod openshift-network-diagnostics/network-check-target-8gs2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:13.223445 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:13.223307    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz podName:50270559-ac36-4cd1-85e4-1267baf1b3cc nodeName:}" failed. No retries permitted until 2026-09-17 23:11:17.223283188 +0000 UTC m=+10.267659332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6c8dz" (UniqueName: "kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz") pod "network-check-target-8gs2j" (UID: "50270559-ac36-4cd1-85e4-1267baf1b3cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:13.497573 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:13.496779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:13.497573 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:13.496927    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:13.497573 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:13.497352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:13.497573 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:13.497451    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:13.498438 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:13.498200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:13.498438 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:13.498331    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:15.499728 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:15.498873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:15.499728 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:15.499018    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:15.499728 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:15.499402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:15.499728 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:15.499503    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:15.499728 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:15.499554    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:15.499728 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:15.499644    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:17.061060 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:17.061019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:17.061587 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:17.061124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:17.061587 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:17.061229    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:17.061587 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:17.061277    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret podName:0bc41eaf-0a42-4187-8eec-c27fd3ab26d8 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:25.061264202 +0000 UTC m=+18.105640341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret") pod "global-pull-secret-syncer-xt5fb" (UID: "0bc41eaf-0a42-4187-8eec-c27fd3ab26d8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:17.061796 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:17.061647    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:17.061796 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:17.061710    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs podName:1e6da33b-9d1e-472a-97aa-e4951d3972ff nodeName:}" failed. No retries permitted until 2026-09-17 23:11:25.061676421 +0000 UTC m=+18.106052560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs") pod "network-metrics-daemon-gr7p6" (UID: "1e6da33b-9d1e-472a-97aa-e4951d3972ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:17.262308 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:17.262268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8dz\" (UniqueName: \"kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz\") pod \"network-check-target-8gs2j\" (UID: \"50270559-ac36-4cd1-85e4-1267baf1b3cc\") " pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:17.262503 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:17.262463    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:17.262503 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:17.262487    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:17.262503 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:17.262501    2580 projected.go:196] Error preparing data for projected volume kube-api-access-6c8dz for pod openshift-network-diagnostics/network-check-target-8gs2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:17.262677 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:17.262563    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz podName:50270559-ac36-4cd1-85e4-1267baf1b3cc nodeName:}" failed. No retries permitted until 2026-09-17 23:11:25.262543335 +0000 UTC m=+18.306919480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6c8dz" (UniqueName: "kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz") pod "network-check-target-8gs2j" (UID: "50270559-ac36-4cd1-85e4-1267baf1b3cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:17.496907 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:17.496831    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:17.497077 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:17.496955    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:17.497077 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:17.497045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:17.497196 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:17.497092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:17.497196 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:17.497177    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:17.497307 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:17.497275    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:19.496185 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:19.496001    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:19.496185 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:19.496023    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:19.496185 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:19.496131    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:19.496763 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:19.496284    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:19.496763 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:19.496418    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:19.496763 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:19.496519    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:21.501015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:21.500984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:21.501437 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:21.501087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:21.501437 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:21.501101    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:21.501437 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:21.501121    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:21.501437 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:21.501212    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:21.501437 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:21.501280    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:23.496554 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:23.496516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:23.497026 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:23.496516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:23.497026 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:23.496690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:23.497026 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:23.496682    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:23.497026 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:23.496813    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:23.497026 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:23.496900    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:25.122352 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:25.122325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:25.122782 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:25.122382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:25.122782 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:25.122465    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:25.122782 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:25.122473    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:25.122782 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:25.122514    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret podName:0bc41eaf-0a42-4187-8eec-c27fd3ab26d8 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:41.122502124 +0000 UTC m=+34.166878262 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret") pod "global-pull-secret-syncer-xt5fb" (UID: "0bc41eaf-0a42-4187-8eec-c27fd3ab26d8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:25.122782 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:25.122526    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs podName:1e6da33b-9d1e-472a-97aa-e4951d3972ff nodeName:}" failed. No retries permitted until 2026-09-17 23:11:41.122520477 +0000 UTC m=+34.166896614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs") pod "network-metrics-daemon-gr7p6" (UID: "1e6da33b-9d1e-472a-97aa-e4951d3972ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:25.324376 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:25.324340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8dz\" (UniqueName: \"kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz\") pod \"network-check-target-8gs2j\" (UID: \"50270559-ac36-4cd1-85e4-1267baf1b3cc\") " pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:25.324526 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:25.324453    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:25.324526 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:25.324468    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:25.324526 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:25.324477    2580 projected.go:196] Error preparing data for projected volume kube-api-access-6c8dz for pod openshift-network-diagnostics/network-check-target-8gs2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:25.324526 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:25.324522    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz podName:50270559-ac36-4cd1-85e4-1267baf1b3cc nodeName:}" failed. No retries permitted until 2026-09-17 23:11:41.324510226 +0000 UTC m=+34.368886363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6c8dz" (UniqueName: "kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz") pod "network-check-target-8gs2j" (UID: "50270559-ac36-4cd1-85e4-1267baf1b3cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:25.496576 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:25.496498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:25.496576 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:25.496520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:25.496803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:25.496498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:25.496803 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:25.496631    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:25.496803 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:25.496715    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:25.496954 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:25.496801    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:27.497274 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:27.497240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:27.497722 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:27.497326    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:27.497722 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:27.497361    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:27.497722 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:27.497408    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:27.497722 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:27.497419    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:27.497722 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:27.497481    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:29.496765 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.496459    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:29.497172 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.496473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:29.497172 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:29.496851    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:29.497172 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.496511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:29.497172 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:29.496914    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:29.497172 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:29.497005    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:29.758472 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.758437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp" event={"ID":"bbd99a55-d6c1-44c0-adda-255aab5fb532","Type":"ContainerStarted","Data":"f81ba68c651f15e37825d7d5e0e7e42492de6d6074b9f7d38d1151d466260e0e"}
Sep 17 23:11:29.758661 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.758530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp" event={"ID":"bbd99a55-d6c1-44c0-adda-255aab5fb532","Type":"ContainerStarted","Data":"b2db7dc3bae3a063de06eb2de3ff016d41e67dc6d3a27d054cd13a3fa8b4c005"}
Sep 17 23:11:29.758661 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.758574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp" event={"ID":"bbd99a55-d6c1-44c0-adda-255aab5fb532","Type":"ContainerStarted","Data":"3dd966a47fe0ba687d1f5a5b463dad228b83778d4e2aaf0e64ec81f4b4643055"}
Sep 17 23:11:29.758661 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.758633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp" event={"ID":"bbd99a55-d6c1-44c0-adda-255aab5fb532","Type":"ContainerStarted","Data":"a7d5031de584fa44aefd64eea8b44723cf2844e0d5407176be0569ee6a7a1f94"}
Sep 17 23:11:29.758811 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.758671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp" event={"ID":"bbd99a55-d6c1-44c0-adda-255aab5fb532","Type":"ContainerStarted","Data":"4ebe4212be4dd13815302d522dd858876408a8b307d0889350d0c48c53f77cff"}
Sep 17 23:11:29.758811 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.758702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp" event={"ID":"bbd99a55-d6c1-44c0-adda-255aab5fb532","Type":"ContainerStarted","Data":"f2e26a7db5769de1f39e04bc359a861fbd17f836a95816c15bb0de0c2b86e0e9"}
Sep 17 23:11:29.760032 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.760004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zgnlk" event={"ID":"ff87e86b-3f0f-4ab5-ba79-86dae21e2c95","Type":"ContainerStarted","Data":"70827c933cda0e1a756aaf4f8be70aacf3bb692965c7a275afd6c6f6a34fcadd"}
Sep 17 23:11:29.761574 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.761548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kcvpd" event={"ID":"265e74c8-a31a-43de-af18-711758cc8e5d","Type":"ContainerStarted","Data":"5d0d3932d68799784619c434c52f9e75e2723fd3c26135f73e8af26f7fdb8846"}
Sep 17 23:11:29.763098 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.763071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t65zf" event={"ID":"60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5","Type":"ContainerStarted","Data":"be033db2d150d6109e28f3523e01f2b1f97152f36536922bc1cbe92b51ddad37"}
Sep 17 23:11:29.764923 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.764893    2580 generic.go:356] "Generic (PLEG): container finished" podID="405ad021-9316-4805-b71b-3a6a13854a1d" containerID="3d09ce71ee355a83db9990ea41ba0bc07e6ac947552fc5df5703176aa8ccb686" exitCode=0
Sep 17 23:11:29.765139 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.765122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2p4f8" event={"ID":"405ad021-9316-4805-b71b-3a6a13854a1d","Type":"ContainerDied","Data":"3d09ce71ee355a83db9990ea41ba0bc07e6ac947552fc5df5703176aa8ccb686"}
Sep 17 23:11:29.766840 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.766820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xdzcp" event={"ID":"138a1459-9d12-4a9a-8d9d-b34189a27764","Type":"ContainerStarted","Data":"8a54822ebf31252e0b3e8ea66cde912cec0304f73508f8b4f67c759d75bab858"}
Sep 17 23:11:29.768564 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.768538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pgkdk" event={"ID":"70431036-0857-4c0f-83f5-842c5b84a114","Type":"ContainerStarted","Data":"3b054d27d069c8c532e83aa57a4c696225783b4839e8724510f6dd41372ff337"}
Sep 17 23:11:29.770086 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.770061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5" event={"ID":"3d06769a-458c-4280-805b-c24da91a066a","Type":"ContainerStarted","Data":"85eb9ddb6f4c7b8515f5028693eec135c39096662b3c925f30359b3b8d02d794"}
Sep 17 23:11:29.778050 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.777987    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zgnlk" podStartSLOduration=3.280218531 podStartE2EDuration="21.777961683s" totalImagesPullingTime="18.497743152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:29.777005894 +0000 UTC m=+22.821382053" watchObservedRunningTime="2026-09-17 23:11:29.777961683 +0000 UTC m=+22.822337840"
Sep 17 23:11:29.778727 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.778691    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-184.ec2.internal" podStartSLOduration=20.778675879 podStartE2EDuration="20.778675879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:12.73417887 +0000 UTC m=+5.778555034" watchObservedRunningTime="2026-09-17 23:11:29.778675879 +0000 UTC m=+22.823052039"
Sep 17 23:11:29.793940 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.793895    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pgkdk" podStartSLOduration=10.505722841 podStartE2EDuration="21.791440469s" totalImagesPullingTime="11.285717628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:29.790843334 +0000 UTC m=+22.835219487" watchObservedRunningTime="2026-09-17 23:11:29.791440469 +0000 UTC m=+22.835816630"
Sep 17 23:11:29.829127 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.829080    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xdzcp" podStartSLOduration=3.384984745 podStartE2EDuration="21.829066233s" totalImagesPullingTime="18.444081488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:29.828965708 +0000 UTC m=+22.873341867" watchObservedRunningTime="2026-09-17 23:11:29.829066233 +0000 UTC m=+22.873442392"
Sep 17 23:11:29.846361 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:29.846322    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kcvpd" podStartSLOduration=3.409850432 podStartE2EDuration="21.846308658s" totalImagesPullingTime="18.436458226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:29.845963501 +0000 UTC m=+22.890339672" watchObservedRunningTime="2026-09-17 23:11:29.846308658 +0000 UTC m=+22.890684816"
Sep 17 23:11:30.159239 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:30.159211    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pgkdk"
Sep 17 23:11:30.159819 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:30.159798    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pgkdk"
Sep 17 23:11:30.175757 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:30.175719    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t65zf" podStartSLOduration=3.722762607 podStartE2EDuration="22.175706824s" totalImagesPullingTime="18.452944217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:29.867000135 +0000 UTC m=+22.911376294" watchObservedRunningTime="2026-09-17 23:11:30.175706824 +0000 UTC m=+23.220082982"
Sep 17 23:11:30.550937 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:30.550913    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 23:11:30.774911 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:30.774808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zn8l4" event={"ID":"39469e70-5457-494d-9f87-7cf3f8c7d557","Type":"ContainerStarted","Data":"29488d386a9ef8be9eb596fa611831a3ffc4d1f09a1156beed1f609f46e62e02"}
Sep 17 23:11:30.776831 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:30.776801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5" event={"ID":"3d06769a-458c-4280-805b-c24da91a066a","Type":"ContainerStarted","Data":"2a69bb3f987db7be3a3a48dea45b1f56d9ed376bbce3dab0c8472b10787b2029"}
Sep 17 23:11:30.777579 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:30.777546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pgkdk"
Sep 17 23:11:30.778063 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:30.778040    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pgkdk"
Sep 17 23:11:30.804115 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:30.804068    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zn8l4" podStartSLOduration=4.364185402 podStartE2EDuration="22.804047984s" totalImagesPullingTime="18.439862582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:30.789182865 +0000 UTC m=+23.833559023" watchObservedRunningTime="2026-09-17 23:11:30.804047984 +0000 UTC m=+23.848424143"
Sep 17 23:11:31.455498 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:31.455406    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:11:30.550931324Z","UUID":"fd268d7f-f75d-4086-b099-e3537f528bf1","Handler":null,"Name":"","Endpoint":""}
Sep 17 23:11:31.457301 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:31.457277    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 23:11:31.457420 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:31.457311    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 23:11:31.495942 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:31.495917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:31.496088 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:31.495917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:31.496088 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:31.496034    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:31.496193 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:31.495917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:31.496193 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:31.496106    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:31.496259 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:31.496203    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:31.781976 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:31.781939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5" event={"ID":"3d06769a-458c-4280-805b-c24da91a066a","Type":"ContainerStarted","Data":"330f09144d48fc86a2cf8ba72892e44f75d5d5a978c631791ac83dfeb8cded17"}
Sep 17 23:11:31.786128 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:31.786103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp" event={"ID":"bbd99a55-d6c1-44c0-adda-255aab5fb532","Type":"ContainerStarted","Data":"39c4e9b1fab291de43204d25b708cb445163643fcdec43a64ee5cfc13a0aed46"}
Sep 17 23:11:32.808593 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:32.808346    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfvl5" podStartSLOduration=4.120119026 podStartE2EDuration="24.808326158s" totalImagesPullingTime="20.688207132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:32.808213032 +0000 UTC m=+25.852589190" watchObservedRunningTime="2026-09-17 23:11:32.808326158 +0000 UTC m=+25.852702315"
Sep 17 23:11:33.495855 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:33.495823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:33.496040 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:33.495832    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:33.496040 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:33.495955    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:33.496040 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:33.495971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:33.496228 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:33.496089    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:33.496228 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:33.496189    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:34.793698 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:34.793661    2580 generic.go:356] "Generic (PLEG): container finished" podID="405ad021-9316-4805-b71b-3a6a13854a1d" containerID="6aa72e11d1cb5e37f577c91fa01a8fd8a04858c51eaee4126c5f75b1ca090bbd" exitCode=0
Sep 17 23:11:34.794267 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:34.793752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2p4f8" event={"ID":"405ad021-9316-4805-b71b-3a6a13854a1d","Type":"ContainerDied","Data":"6aa72e11d1cb5e37f577c91fa01a8fd8a04858c51eaee4126c5f75b1ca090bbd"}
Sep 17 23:11:34.797523 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:34.797488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp" event={"ID":"bbd99a55-d6c1-44c0-adda-255aab5fb532","Type":"ContainerStarted","Data":"06d4337f230aee487a45449085e2ab7daa7408665be2431aa4b03cefd023418b"}
Sep 17 23:11:34.797775 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:34.797756    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:34.797867 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:34.797854    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:34.797913 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:34.797903    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:34.814359 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:34.814334    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:34.814483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:34.814434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:11:34.847350 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:34.847307    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp" podStartSLOduration=8.351699875 podStartE2EDuration="26.847293555s" totalImagesPullingTime="18.49559368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:34.845667688 +0000 UTC m=+27.890043847" watchObservedRunningTime="2026-09-17 23:11:34.847293555 +0000 UTC m=+27.891669713"
Sep 17 23:11:35.496496 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:35.496263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:35.496923 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:35.496340    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:35.496923 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:35.496392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:35.496923 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:35.496627    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:35.496923 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:35.496703    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:35.496923 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:35.496791    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:35.801999 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:35.801973    2580 generic.go:356] "Generic (PLEG): container finished" podID="405ad021-9316-4805-b71b-3a6a13854a1d" containerID="df5eb40e590bdc156e6771e606d1229817ef0aa6a478f48b6c35361f5c593ac1" exitCode=0
Sep 17 23:11:35.802509 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:35.802062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2p4f8" event={"ID":"405ad021-9316-4805-b71b-3a6a13854a1d","Type":"ContainerDied","Data":"df5eb40e590bdc156e6771e606d1229817ef0aa6a478f48b6c35361f5c593ac1"}
Sep 17 23:11:35.808798 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:35.808775    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8gs2j"]
Sep 17 23:11:35.808899 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:35.808854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:35.808969 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:35.808946    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:35.810658 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:35.810637    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xt5fb"]
Sep 17 23:11:35.810741 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:35.810719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:35.810841 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:35.810819    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:35.820164 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:35.820144    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gr7p6"]
Sep 17 23:11:35.820246 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:35.820211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:35.820282 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:35.820273    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:37.497200 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:37.497167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:37.497770 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:37.497249    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:37.497770 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:37.497335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:37.497770 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:37.497412    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:37.497770 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:37.497453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:37.497770 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:37.497499    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:37.808921 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:37.808890    2580 generic.go:356] "Generic (PLEG): container finished" podID="405ad021-9316-4805-b71b-3a6a13854a1d" containerID="c56140abe4d665158e178542ff23f358df998ee0fd21b3302eb4a52988e35947" exitCode=0
Sep 17 23:11:37.809119 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:37.808942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2p4f8" event={"ID":"405ad021-9316-4805-b71b-3a6a13854a1d","Type":"ContainerDied","Data":"c56140abe4d665158e178542ff23f358df998ee0fd21b3302eb4a52988e35947"}
Sep 17 23:11:39.495720 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:39.495693    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:39.496423 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:39.495694    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:39.496423 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:39.495816    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:39.496423 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:39.495694    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:39.496423 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:39.495898    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:39.496423 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:39.496021    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:41.138493 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.138271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:41.138942 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.138526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:41.138942 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:41.138422    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:41.138942 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:41.138668    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret podName:0bc41eaf-0a42-4187-8eec-c27fd3ab26d8 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:13.13864314 +0000 UTC m=+66.183019292 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret") pod "global-pull-secret-syncer-xt5fb" (UID: "0bc41eaf-0a42-4187-8eec-c27fd3ab26d8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:41.138942 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:41.138678    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:41.138942 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:41.138729    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs podName:1e6da33b-9d1e-472a-97aa-e4951d3972ff nodeName:}" failed. No retries permitted until 2026-09-17 23:12:13.138712623 +0000 UTC m=+66.183088776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs") pod "network-metrics-daemon-gr7p6" (UID: "1e6da33b-9d1e-472a-97aa-e4951d3972ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:41.340646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.340595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8dz\" (UniqueName: \"kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz\") pod \"network-check-target-8gs2j\" (UID: \"50270559-ac36-4cd1-85e4-1267baf1b3cc\") " pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:41.340821 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:41.340752    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:41.340821 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:41.340775    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:41.340821 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:41.340786    2580 projected.go:196] Error preparing data for projected volume kube-api-access-6c8dz for pod openshift-network-diagnostics/network-check-target-8gs2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:41.340974 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:41.340848    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz podName:50270559-ac36-4cd1-85e4-1267baf1b3cc nodeName:}" failed. No retries permitted until 2026-09-17 23:12:13.340830506 +0000 UTC m=+66.385206656 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6c8dz" (UniqueName: "kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz") pod "network-check-target-8gs2j" (UID: "50270559-ac36-4cd1-85e4-1267baf1b3cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:41.496385 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.496265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:41.496385 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.496270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:41.496564 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:41.496394    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt5fb" podUID="0bc41eaf-0a42-4187-8eec-c27fd3ab26d8"
Sep 17 23:11:41.496564 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:41.496487    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7p6" podUID="1e6da33b-9d1e-472a-97aa-e4951d3972ff"
Sep 17 23:11:41.496564 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.496270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:41.496697 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:41.496577    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gs2j" podUID="50270559-ac36-4cd1-85e4-1267baf1b3cc"
Sep 17 23:11:41.813284 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.813261    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-184.ec2.internal" event="NodeReady"
Sep 17 23:11:41.813421 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.813410    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 23:11:41.848675 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.848647    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd"]
Sep 17 23:11:41.851786 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.851760    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7986d6c7dd-sk4nw"]
Sep 17 23:11:41.851970 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.851949    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd"
Sep 17 23:11:41.854204 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.854184    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 23:11:41.854360 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.854251    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-l449f\""
Sep 17 23:11:41.854360 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.854288    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 23:11:41.854505 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.854490    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 23:11:41.854563 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.854528    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 23:11:41.854686 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.854571    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"]
Sep 17 23:11:41.854754 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.854715    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:41.857076 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.857056    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 23:11:41.857170 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.857058    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 23:11:41.857170 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.857103    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dcs8c\""
Sep 17 23:11:41.857371 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.857355    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 23:11:41.857418 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.857384    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"]
Sep 17 23:11:41.857672 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.857646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:41.863576 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.861527    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 23:11:41.863576 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.862524    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd"]
Sep 17 23:11:41.863576 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.862685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:41.864347 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.864327    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rj6f4"]
Sep 17 23:11:41.865012 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.864968    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 23:11:41.865483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.865357    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 23:11:41.865483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.865366    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 23:11:41.866370 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.866155    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 23:11:41.867726 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.867708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rj6f4"
Sep 17 23:11:41.868541 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.868523    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"]
Sep 17 23:11:41.868792 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.868747    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 23:11:41.869646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.869492    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7986d6c7dd-sk4nw"]
Sep 17 23:11:41.870476 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.870385    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 23:11:41.870568 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.870505    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x8v6t"]
Sep 17 23:11:41.870653 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.870596    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 23:11:41.870881 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.870864    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pzsn2\""
Sep 17 23:11:41.871067 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.871052    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 23:11:41.873588 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.873569    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:41.875754 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.875736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 23:11:41.875857 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.875772    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 23:11:41.875960 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.875948    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9p4tt\""
Sep 17 23:11:41.880594 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.880572    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rj6f4"]
Sep 17 23:11:41.884866 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.884848    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"]
Sep 17 23:11:41.885583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.885568    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x8v6t"]
Sep 17 23:11:41.945752 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.945317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-ca-trust-extracted\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:41.945752 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.945385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-certificates\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:41.945752 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.945415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2616b995-a9f6-4431-830a-f0a15ac8c2be-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-668d886454-8jhcd\" (UID: \"2616b995-a9f6-4431-830a-f0a15ac8c2be\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd"
Sep 17 23:11:41.945752 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.945457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-image-registry-private-configuration\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:41.945752 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.945483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-trusted-ca\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:41.945752 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.945505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-installation-pull-secrets\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:41.945752 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.945534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhlb\" (UniqueName: \"kubernetes.io/projected/2616b995-a9f6-4431-830a-f0a15ac8c2be-kube-api-access-tqhlb\") pod \"managed-serviceaccount-addon-agent-668d886454-8jhcd\" (UID: \"2616b995-a9f6-4431-830a-f0a15ac8c2be\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd"
Sep 17 23:11:41.945752 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.945556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-bound-sa-token\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:41.945752 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.945669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:41.945752 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:41.945693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdhb\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-kube-api-access-xrdhb\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.046533 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-image-registry-private-configuration\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.046677 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-trusted-ca\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.046677 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-installation-pull-secrets\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.046677 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhlb\" (UniqueName: \"kubernetes.io/projected/2616b995-a9f6-4431-830a-f0a15ac8c2be-kube-api-access-tqhlb\") pod \"managed-serviceaccount-addon-agent-668d886454-8jhcd\" (UID: \"2616b995-a9f6-4431-830a-f0a15ac8c2be\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd"
Sep 17 23:11:42.046677 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-bound-sa-token\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.046677 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-tmp-dir\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:42.046677 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3f113e0e-9447-42c5-9893-d2bbca7c4403-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.046954 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2csf2\" (UniqueName: \"kubernetes.io/projected/11ed4035-5836-4a4a-ab0e-d73776ada4cf-kube-api-access-2csf2\") pod \"ingress-canary-rj6f4\" (UID: \"11ed4035-5836-4a4a-ab0e-d73776ada4cf\") " pod="openshift-ingress-canary/ingress-canary-rj6f4"
Sep 17 23:11:42.046954 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f113e0e-9447-42c5-9893-d2bbca7c4403-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.046954 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/97f53452-6598-4eaa-b1ef-2c01185c52b5-klusterlet-config\") pod \"klusterlet-addon-workmgr-788d5dc4ff-rsl4m\" (UID: \"97f53452-6598-4eaa-b1ef-2c01185c52b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:42.046954 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7525g\" (UniqueName: \"kubernetes.io/projected/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-kube-api-access-7525g\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:42.046954 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11ed4035-5836-4a4a-ab0e-d73776ada4cf-cert\") pod \"ingress-canary-rj6f4\" (UID: \"11ed4035-5836-4a4a-ab0e-d73776ada4cf\") " pod="openshift-ingress-canary/ingress-canary-rj6f4"
Sep 17 23:11:42.046954 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbq8\" (UniqueName: \"kubernetes.io/projected/97f53452-6598-4eaa-b1ef-2c01185c52b5-kube-api-access-gkbq8\") pod \"klusterlet-addon-workmgr-788d5dc4ff-rsl4m\" (UID: \"97f53452-6598-4eaa-b1ef-2c01185c52b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:42.046954 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.046954 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdhb\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-kube-api-access-xrdhb\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.046954 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-ca-trust-extracted\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.046954 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97f53452-6598-4eaa-b1ef-2c01185c52b5-tmp\") pod \"klusterlet-addon-workmgr-788d5dc4ff-rsl4m\" (UID: \"97f53452-6598-4eaa-b1ef-2c01185c52b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:42.047363 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:42.046958    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:42.047363 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.046975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3f113e0e-9447-42c5-9893-d2bbca7c4403-ca\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.047363 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:42.046980    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7986d6c7dd-sk4nw: secret "image-registry-tls" not found
Sep 17 23:11:42.047363 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.047027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3f113e0e-9447-42c5-9893-d2bbca7c4403-hub\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.047363 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:42.047048    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls podName:ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a nodeName:}" failed. No retries permitted until 2026-09-17 23:11:42.547027007 +0000 UTC m=+35.591403146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls") pod "image-registry-7986d6c7dd-sk4nw" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a") : secret "image-registry-tls" not found
Sep 17 23:11:42.047363 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.047094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-certificates\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.047363 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.047121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-config-volume\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:42.047363 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.047163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:42.047363 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.047196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2616b995-a9f6-4431-830a-f0a15ac8c2be-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-668d886454-8jhcd\" (UID: \"2616b995-a9f6-4431-830a-f0a15ac8c2be\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd"
Sep 17 23:11:42.047363 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.047218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f113e0e-9447-42c5-9893-d2bbca7c4403-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.047363 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.047246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68p8c\" (UniqueName: \"kubernetes.io/projected/3f113e0e-9447-42c5-9893-d2bbca7c4403-kube-api-access-68p8c\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.047363 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.047282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-ca-trust-extracted\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.047862 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.047477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-trusted-ca\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.047862 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.047593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-certificates\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.050700 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.050679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-installation-pull-secrets\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.050793 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.050678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-image-registry-private-configuration\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.050793 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.050753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2616b995-a9f6-4431-830a-f0a15ac8c2be-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-668d886454-8jhcd\" (UID: \"2616b995-a9f6-4431-830a-f0a15ac8c2be\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd"
Sep 17 23:11:42.056512 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.056487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-bound-sa-token\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.057328 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.057308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhlb\" (UniqueName: \"kubernetes.io/projected/2616b995-a9f6-4431-830a-f0a15ac8c2be-kube-api-access-tqhlb\") pod \"managed-serviceaccount-addon-agent-668d886454-8jhcd\" (UID: \"2616b995-a9f6-4431-830a-f0a15ac8c2be\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd"
Sep 17 23:11:42.057414 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.057394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdhb\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-kube-api-access-xrdhb\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.148315 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3f113e0e-9447-42c5-9893-d2bbca7c4403-hub\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.148570 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-config-volume\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:42.148570 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:42.148570 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f113e0e-9447-42c5-9893-d2bbca7c4403-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.148570 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:42.148417    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:42.148570 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:42.148473    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls podName:6be6dc27-bc71-4d36-a57f-7f3b8557d0c3 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:42.648456547 +0000 UTC m=+35.692832700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls") pod "dns-default-x8v6t" (UID: "6be6dc27-bc71-4d36-a57f-7f3b8557d0c3") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:42.148570 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68p8c\" (UniqueName: \"kubernetes.io/projected/3f113e0e-9447-42c5-9893-d2bbca7c4403-kube-api-access-68p8c\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.148570 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-tmp-dir\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:42.148991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3f113e0e-9447-42c5-9893-d2bbca7c4403-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.148991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2csf2\" (UniqueName: \"kubernetes.io/projected/11ed4035-5836-4a4a-ab0e-d73776ada4cf-kube-api-access-2csf2\") pod \"ingress-canary-rj6f4\" (UID: \"11ed4035-5836-4a4a-ab0e-d73776ada4cf\") " pod="openshift-ingress-canary/ingress-canary-rj6f4"
Sep 17 23:11:42.148991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f113e0e-9447-42c5-9893-d2bbca7c4403-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.148991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/97f53452-6598-4eaa-b1ef-2c01185c52b5-klusterlet-config\") pod \"klusterlet-addon-workmgr-788d5dc4ff-rsl4m\" (UID: \"97f53452-6598-4eaa-b1ef-2c01185c52b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:42.148991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7525g\" (UniqueName: \"kubernetes.io/projected/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-kube-api-access-7525g\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:42.148991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11ed4035-5836-4a4a-ab0e-d73776ada4cf-cert\") pod \"ingress-canary-rj6f4\" (UID: \"11ed4035-5836-4a4a-ab0e-d73776ada4cf\") " pod="openshift-ingress-canary/ingress-canary-rj6f4"
Sep 17 23:11:42.148991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbq8\" (UniqueName: \"kubernetes.io/projected/97f53452-6598-4eaa-b1ef-2c01185c52b5-kube-api-access-gkbq8\") pod \"klusterlet-addon-workmgr-788d5dc4ff-rsl4m\" (UID: \"97f53452-6598-4eaa-b1ef-2c01185c52b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:42.148991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97f53452-6598-4eaa-b1ef-2c01185c52b5-tmp\") pod \"klusterlet-addon-workmgr-788d5dc4ff-rsl4m\" (UID: \"97f53452-6598-4eaa-b1ef-2c01185c52b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:42.148991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3f113e0e-9447-42c5-9893-d2bbca7c4403-ca\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.148991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.148878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-tmp-dir\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:42.149463 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.149074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-config-volume\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:42.150087 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.149690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f113e0e-9447-42c5-9893-d2bbca7c4403-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.150087 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.150035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97f53452-6598-4eaa-b1ef-2c01185c52b5-tmp\") pod \"klusterlet-addon-workmgr-788d5dc4ff-rsl4m\" (UID: \"97f53452-6598-4eaa-b1ef-2c01185c52b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:42.151160 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.151139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3f113e0e-9447-42c5-9893-d2bbca7c4403-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.151257 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.151244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3f113e0e-9447-42c5-9893-d2bbca7c4403-hub\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.151454 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.151431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3f113e0e-9447-42c5-9893-d2bbca7c4403-ca\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.151536 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.151435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f113e0e-9447-42c5-9893-d2bbca7c4403-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.151668 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.151654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/97f53452-6598-4eaa-b1ef-2c01185c52b5-klusterlet-config\") pod \"klusterlet-addon-workmgr-788d5dc4ff-rsl4m\" (UID: \"97f53452-6598-4eaa-b1ef-2c01185c52b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:42.151710 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.151692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11ed4035-5836-4a4a-ab0e-d73776ada4cf-cert\") pod \"ingress-canary-rj6f4\" (UID: \"11ed4035-5836-4a4a-ab0e-d73776ada4cf\") " pod="openshift-ingress-canary/ingress-canary-rj6f4"
Sep 17 23:11:42.158780 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.158760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7525g\" (UniqueName: \"kubernetes.io/projected/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-kube-api-access-7525g\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:42.158935 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.158912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68p8c\" (UniqueName: \"kubernetes.io/projected/3f113e0e-9447-42c5-9893-d2bbca7c4403-kube-api-access-68p8c\") pod \"cluster-proxy-proxy-agent-65dffc8776-6fngw\" (UID: \"3f113e0e-9447-42c5-9893-d2bbca7c4403\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.159540 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.159516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2csf2\" (UniqueName: \"kubernetes.io/projected/11ed4035-5836-4a4a-ab0e-d73776ada4cf-kube-api-access-2csf2\") pod \"ingress-canary-rj6f4\" (UID: \"11ed4035-5836-4a4a-ab0e-d73776ada4cf\") " pod="openshift-ingress-canary/ingress-canary-rj6f4"
Sep 17 23:11:42.159745 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.159729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbq8\" (UniqueName: \"kubernetes.io/projected/97f53452-6598-4eaa-b1ef-2c01185c52b5-kube-api-access-gkbq8\") pod \"klusterlet-addon-workmgr-788d5dc4ff-rsl4m\" (UID: \"97f53452-6598-4eaa-b1ef-2c01185c52b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:42.174557 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.174538    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd"
Sep 17 23:11:42.188995 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.188974    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:42.195501 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.195247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:11:42.209708 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.209689    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rj6f4"
Sep 17 23:11:42.457631 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.457552    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd"]
Sep 17 23:11:42.460623 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:42.460580    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2616b995_a9f6_4431_830a_f0a15ac8c2be.slice/crio-9f141d84971db5115c3b5aa0c1b1510a3ead78e94af6c7b4793d2a8851773c36 WatchSource:0}: Error finding container 9f141d84971db5115c3b5aa0c1b1510a3ead78e94af6c7b4793d2a8851773c36: Status 404 returned error can't find the container with id 9f141d84971db5115c3b5aa0c1b1510a3ead78e94af6c7b4793d2a8851773c36
Sep 17 23:11:42.465002 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.464833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"]
Sep 17 23:11:42.469051 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:42.469024    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f53452_6598_4eaa_b1ef_2c01185c52b5.slice/crio-fbcf12114575146a9d66356bdf8470ebf5e6abd52b5c598056bc91e8f9fb1642 WatchSource:0}: Error finding container fbcf12114575146a9d66356bdf8470ebf5e6abd52b5c598056bc91e8f9fb1642: Status 404 returned error can't find the container with id fbcf12114575146a9d66356bdf8470ebf5e6abd52b5c598056bc91e8f9fb1642
Sep 17 23:11:42.491454 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.491431    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"]
Sep 17 23:11:42.493648 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:42.493628    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f113e0e_9447_42c5_9893_d2bbca7c4403.slice/crio-16e423150f37f051afc128c4be400f01fe0ebba3cb963a44b73b1505a35f5404 WatchSource:0}: Error finding container 16e423150f37f051afc128c4be400f01fe0ebba3cb963a44b73b1505a35f5404: Status 404 returned error can't find the container with id 16e423150f37f051afc128c4be400f01fe0ebba3cb963a44b73b1505a35f5404
Sep 17 23:11:42.502841 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.502822    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rj6f4"]
Sep 17 23:11:42.505352 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:11:42.505332    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ed4035_5836_4a4a_ab0e_d73776ada4cf.slice/crio-10d789bfd2ff5f4fbcff0c3e738b4e104415d5884badc8158805e234252e058e WatchSource:0}: Error finding container 10d789bfd2ff5f4fbcff0c3e738b4e104415d5884badc8158805e234252e058e: Status 404 returned error can't find the container with id 10d789bfd2ff5f4fbcff0c3e738b4e104415d5884badc8158805e234252e058e
Sep 17 23:11:42.552435 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.552410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:42.552527 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:42.552516    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:42.552563 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:42.552529    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7986d6c7dd-sk4nw: secret "image-registry-tls" not found
Sep 17 23:11:42.552595 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:42.552567    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls podName:ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a nodeName:}" failed. No retries permitted until 2026-09-17 23:11:43.55255534 +0000 UTC m=+36.596931478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls") pod "image-registry-7986d6c7dd-sk4nw" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a") : secret "image-registry-tls" not found
Sep 17 23:11:42.653350 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.653328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:42.653436 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:42.653418    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:42.653471 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:42.653466    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls podName:6be6dc27-bc71-4d36-a57f-7f3b8557d0c3 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:43.65345113 +0000 UTC m=+36.697827269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls") pod "dns-default-x8v6t" (UID: "6be6dc27-bc71-4d36-a57f-7f3b8557d0c3") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:42.823789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.823761    2580 generic.go:356] "Generic (PLEG): container finished" podID="405ad021-9316-4805-b71b-3a6a13854a1d" containerID="c97cb079adffd4391758f91edcdde3c9a3ed3c8332b644b8a14246488234a5b1" exitCode=0
Sep 17 23:11:42.823956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.823847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2p4f8" event={"ID":"405ad021-9316-4805-b71b-3a6a13854a1d","Type":"ContainerDied","Data":"c97cb079adffd4391758f91edcdde3c9a3ed3c8332b644b8a14246488234a5b1"}
Sep 17 23:11:42.825171 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.824977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rj6f4" event={"ID":"11ed4035-5836-4a4a-ab0e-d73776ada4cf","Type":"ContainerStarted","Data":"10d789bfd2ff5f4fbcff0c3e738b4e104415d5884badc8158805e234252e058e"}
Sep 17 23:11:42.826195 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.826170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m" event={"ID":"97f53452-6598-4eaa-b1ef-2c01185c52b5","Type":"ContainerStarted","Data":"fbcf12114575146a9d66356bdf8470ebf5e6abd52b5c598056bc91e8f9fb1642"}
Sep 17 23:11:42.827364 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.827342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd" event={"ID":"2616b995-a9f6-4431-830a-f0a15ac8c2be","Type":"ContainerStarted","Data":"9f141d84971db5115c3b5aa0c1b1510a3ead78e94af6c7b4793d2a8851773c36"}
Sep 17 23:11:42.828570 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:42.828548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw" event={"ID":"3f113e0e-9447-42c5-9893-d2bbca7c4403","Type":"ContainerStarted","Data":"16e423150f37f051afc128c4be400f01fe0ebba3cb963a44b73b1505a35f5404"}
Sep 17 23:11:43.497645 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.496335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:11:43.497645 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.496919    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:11:43.497645 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.497275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:11:43.500340 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.499162    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:43.500340 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.499376    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:43.500340 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.499933    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:43.500554 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.500408    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dcj7l\""
Sep 17 23:11:43.500640 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.500590    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zzvsp\""
Sep 17 23:11:43.500640 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.500605    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:43.560311 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.560271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:43.560522 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:43.560476    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:43.560522 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:43.560497    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7986d6c7dd-sk4nw: secret "image-registry-tls" not found
Sep 17 23:11:43.560794 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:43.560565    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls podName:ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a nodeName:}" failed. No retries permitted until 2026-09-17 23:11:45.560544062 +0000 UTC m=+38.604920213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls") pod "image-registry-7986d6c7dd-sk4nw" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a") : secret "image-registry-tls" not found
Sep 17 23:11:43.661483 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.661449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:43.661687 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:43.661670    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:43.661760 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:43.661736    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls podName:6be6dc27-bc71-4d36-a57f-7f3b8557d0c3 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:45.661717269 +0000 UTC m=+38.706093427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls") pod "dns-default-x8v6t" (UID: "6be6dc27-bc71-4d36-a57f-7f3b8557d0c3") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:43.847255 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.846419    2580 generic.go:356] "Generic (PLEG): container finished" podID="405ad021-9316-4805-b71b-3a6a13854a1d" containerID="9e306da8df5ae3704ea40a1b3a3bd78be2b2bdb2190375430c102467b09ccaf6" exitCode=0
Sep 17 23:11:43.847255 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:43.846493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2p4f8" event={"ID":"405ad021-9316-4805-b71b-3a6a13854a1d","Type":"ContainerDied","Data":"9e306da8df5ae3704ea40a1b3a3bd78be2b2bdb2190375430c102467b09ccaf6"}
Sep 17 23:11:44.860229 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:44.860132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2p4f8" event={"ID":"405ad021-9316-4805-b71b-3a6a13854a1d","Type":"ContainerStarted","Data":"bd3d8a10c2dcbac23848fd278d5e9ba4d84519ec4635a15d4758f50a4f0a37f3"}
Sep 17 23:11:44.886470 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:44.885202    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2p4f8" podStartSLOduration=8.402821755 podStartE2EDuration="36.885180418s" totalImagesPullingTime="28.482358663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:44.884231921 +0000 UTC m=+37.928608081" watchObservedRunningTime="2026-09-17 23:11:44.885180418 +0000 UTC m=+37.929556572"
Sep 17 23:11:45.581333 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:45.581293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:45.581516 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:45.581465    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:45.581516 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:45.581481    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7986d6c7dd-sk4nw: secret "image-registry-tls" not found
Sep 17 23:11:45.581650 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:45.581606    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls podName:ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a nodeName:}" failed. No retries permitted until 2026-09-17 23:11:49.581587323 +0000 UTC m=+42.625963474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls") pod "image-registry-7986d6c7dd-sk4nw" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a") : secret "image-registry-tls" not found
Sep 17 23:11:45.682766 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:45.682140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:45.682766 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:45.682303    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:45.682766 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:45.682362    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls podName:6be6dc27-bc71-4d36-a57f-7f3b8557d0c3 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:49.682345373 +0000 UTC m=+42.726721512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls") pod "dns-default-x8v6t" (UID: "6be6dc27-bc71-4d36-a57f-7f3b8557d0c3") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:49.607838 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:49.607800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:49.608218 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:49.607950    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:49.608218 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:49.607970    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7986d6c7dd-sk4nw: secret "image-registry-tls" not found
Sep 17 23:11:49.608218 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:49.608024    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls podName:ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a nodeName:}" failed. No retries permitted until 2026-09-17 23:11:57.608008895 +0000 UTC m=+50.652385032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls") pod "image-registry-7986d6c7dd-sk4nw" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a") : secret "image-registry-tls" not found
Sep 17 23:11:49.708991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:49.708961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:49.709107 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:49.709075    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:49.709161 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:49.709141    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls podName:6be6dc27-bc71-4d36-a57f-7f3b8557d0c3 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:57.709122616 +0000 UTC m=+50.753498753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls") pod "dns-default-x8v6t" (UID: "6be6dc27-bc71-4d36-a57f-7f3b8557d0c3") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:51.884426 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:51.884386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rj6f4" event={"ID":"11ed4035-5836-4a4a-ab0e-d73776ada4cf","Type":"ContainerStarted","Data":"0c58cd3afcceb1070477af032e5c401aefd28c71949a8f2ab112c7b17ee4177c"}
Sep 17 23:11:51.885864 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:51.885840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m" event={"ID":"97f53452-6598-4eaa-b1ef-2c01185c52b5","Type":"ContainerStarted","Data":"06e0ac8e1fcdcade2aebedffa53f8764f70358cb11492934620fc1072fdd6eb0"}
Sep 17 23:11:51.885963 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:51.885885    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:51.887402 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:51.887349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd" event={"ID":"2616b995-a9f6-4431-830a-f0a15ac8c2be","Type":"ContainerStarted","Data":"6748c96ee927a3beeff2dc0b15dfbb9dc421b0fafd7fbe60e970c82e5a3190d3"}
Sep 17 23:11:51.887821 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:51.887765    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m"
Sep 17 23:11:51.889357 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:51.889336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw" event={"ID":"3f113e0e-9447-42c5-9893-d2bbca7c4403","Type":"ContainerStarted","Data":"20e18a00541fa08a872c703c542e80de16fdf364f4408d22a3394f8f041d8566"}
Sep 17 23:11:51.889450 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:51.889362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw" event={"ID":"3f113e0e-9447-42c5-9893-d2bbca7c4403","Type":"ContainerStarted","Data":"2195ed5dbf6c5e88e17790ea14864ec9a6961ef0d4af76e5f50a57dd2b0ac809"}
Sep 17 23:11:51.889450 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:51.889378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw" event={"ID":"3f113e0e-9447-42c5-9893-d2bbca7c4403","Type":"ContainerStarted","Data":"b2fdba8e48b172b731e3b69dffe328d18d57c2d8f565027faeb274e71bef4a00"}
Sep 17 23:11:51.900676 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:51.900641    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rj6f4" podStartSLOduration=2.252713668 podStartE2EDuration="10.900627503s" totalImagesPullingTime="8.647913835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:51.900214227 +0000 UTC m=+44.944590385" watchObservedRunningTime="2026-09-17 23:11:51.900627503 +0000 UTC m=+44.945003654"
Sep 17 23:11:51.917395 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:51.917356    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668d886454-8jhcd" podStartSLOduration=22.224658561 podStartE2EDuration="30.917344824s" totalImagesPullingTime="8.692686263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:51.916779192 +0000 UTC m=+44.961155353" watchObservedRunningTime="2026-09-17 23:11:51.917344824 +0000 UTC m=+44.961720982"
Sep 17 23:11:51.935768 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:51.935733    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw" podStartSLOduration=22.24295317 podStartE2EDuration="30.935696317s" totalImagesPullingTime="8.692743147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:51.935012236 +0000 UTC m=+44.979388395" watchObservedRunningTime="2026-09-17 23:11:51.935696317 +0000 UTC m=+44.980072476"
Sep 17 23:11:51.955446 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:51.955399    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788d5dc4ff-rsl4m" podStartSLOduration=22.253409571 podStartE2EDuration="30.955388614s" totalImagesPullingTime="8.701979043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:51.954940095 +0000 UTC m=+44.999316254" watchObservedRunningTime="2026-09-17 23:11:51.955388614 +0000 UTC m=+44.999764769"
Sep 17 23:11:57.663934 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:57.663887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:11:57.664408 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:57.664021    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:57.664408 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:57.664033    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7986d6c7dd-sk4nw: secret "image-registry-tls" not found
Sep 17 23:11:57.664408 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:57.664095    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls podName:ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a nodeName:}" failed. No retries permitted until 2026-09-17 23:12:13.664069839 +0000 UTC m=+66.708445976 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls") pod "image-registry-7986d6c7dd-sk4nw" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a") : secret "image-registry-tls" not found
Sep 17 23:11:57.764775 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:11:57.764740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:11:57.764918 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:57.764890    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:57.764974 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:11:57.764963    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls podName:6be6dc27-bc71-4d36-a57f-7f3b8557d0c3 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:13.764945366 +0000 UTC m=+66.809321504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls") pod "dns-default-x8v6t" (UID: "6be6dc27-bc71-4d36-a57f-7f3b8557d0c3") : secret "dns-default-metrics-tls" not found
Sep 17 23:12:06.819485 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:06.819458    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jnsbp"
Sep 17 23:12:13.178199 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.178160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:12:13.178592 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.178230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:12:13.180885 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.180865    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:12:13.180932 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.180920    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:12:13.188984 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:13.188970    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 23:12:13.189039 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:13.189029    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs podName:1e6da33b-9d1e-472a-97aa-e4951d3972ff nodeName:}" failed. No retries permitted until 2026-09-17 23:13:17.189014419 +0000 UTC m=+130.233390556 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs") pod "network-metrics-daemon-gr7p6" (UID: "1e6da33b-9d1e-472a-97aa-e4951d3972ff") : secret "metrics-daemon-secret" not found
Sep 17 23:12:13.192451 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.192421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bc41eaf-0a42-4187-8eec-c27fd3ab26d8-original-pull-secret\") pod \"global-pull-secret-syncer-xt5fb\" (UID: \"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8\") " pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:12:13.231695 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.231668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt5fb"
Sep 17 23:12:13.380567 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.380526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8dz\" (UniqueName: \"kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz\") pod \"network-check-target-8gs2j\" (UID: \"50270559-ac36-4cd1-85e4-1267baf1b3cc\") " pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:12:13.383450 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.383432    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:12:13.393583 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.393562    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:12:13.404027 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.403999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8dz\" (UniqueName: \"kubernetes.io/projected/50270559-ac36-4cd1-85e4-1267baf1b3cc-kube-api-access-6c8dz\") pod \"network-check-target-8gs2j\" (UID: \"50270559-ac36-4cd1-85e4-1267baf1b3cc\") " pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:12:13.516231 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.516171    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dcj7l\""
Sep 17 23:12:13.522459 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.522436    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xt5fb"]
Sep 17 23:12:13.524608 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.524587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:12:13.525432 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:12:13.525410    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc41eaf_0a42_4187_8eec_c27fd3ab26d8.slice/crio-c72dee281abde8014ca0c3c49e19d9fcd8c52f73860fbfa9e4b82eeabd1ab013 WatchSource:0}: Error finding container c72dee281abde8014ca0c3c49e19d9fcd8c52f73860fbfa9e4b82eeabd1ab013: Status 404 returned error can't find the container with id c72dee281abde8014ca0c3c49e19d9fcd8c52f73860fbfa9e4b82eeabd1ab013
Sep 17 23:12:13.684000 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.683969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:12:13.684168 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:13.684122    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:13.684168 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:13.684141    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7986d6c7dd-sk4nw: secret "image-registry-tls" not found
Sep 17 23:12:13.684265 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:13.684201    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls podName:ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a nodeName:}" failed. No retries permitted until 2026-09-17 23:12:45.684186172 +0000 UTC m=+98.728562310 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls") pod "image-registry-7986d6c7dd-sk4nw" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a") : secret "image-registry-tls" not found
Sep 17 23:12:13.784700 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.784670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:12:13.784821 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:13.784794    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:12:13.784867 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:13.784863    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls podName:6be6dc27-bc71-4d36-a57f-7f3b8557d0c3 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:45.784839546 +0000 UTC m=+98.829215683 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls") pod "dns-default-x8v6t" (UID: "6be6dc27-bc71-4d36-a57f-7f3b8557d0c3") : secret "dns-default-metrics-tls" not found
Sep 17 23:12:13.821932 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.821898    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8gs2j"]
Sep 17 23:12:13.824628 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:12:13.824588    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50270559_ac36_4cd1_85e4_1267baf1b3cc.slice/crio-2f7d9ef1a21d86aa37114073a43563c12c770081c5b142b3cc051729d2ce62c9 WatchSource:0}: Error finding container 2f7d9ef1a21d86aa37114073a43563c12c770081c5b142b3cc051729d2ce62c9: Status 404 returned error can't find the container with id 2f7d9ef1a21d86aa37114073a43563c12c770081c5b142b3cc051729d2ce62c9
Sep 17 23:12:13.954704 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.954662    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xt5fb" event={"ID":"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8","Type":"ContainerStarted","Data":"c72dee281abde8014ca0c3c49e19d9fcd8c52f73860fbfa9e4b82eeabd1ab013"}
Sep 17 23:12:13.956056 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:13.956015    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8gs2j" event={"ID":"50270559-ac36-4cd1-85e4-1267baf1b3cc","Type":"ContainerStarted","Data":"2f7d9ef1a21d86aa37114073a43563c12c770081c5b142b3cc051729d2ce62c9"}
Sep 17 23:12:18.977428 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:18.977391    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xt5fb" event={"ID":"0bc41eaf-0a42-4187-8eec-c27fd3ab26d8","Type":"ContainerStarted","Data":"e0fe9c7832e844ff7b075b57d1d4965db150d5cdbf1a534e7d2a2b8f08db6d15"}
Sep 17 23:12:18.978817 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:18.978794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8gs2j" event={"ID":"50270559-ac36-4cd1-85e4-1267baf1b3cc","Type":"ContainerStarted","Data":"8ca782479770db6a20b289788b2ef0a8abefcffa30a9372e44f08a593c82522c"}
Sep 17 23:12:18.978934 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:18.978919    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:12:18.994093 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:18.994047    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xt5fb" podStartSLOduration=66.420770754 podStartE2EDuration="1m10.994033068s" totalImagesPullingTime="4.573262314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:18.993077671 +0000 UTC m=+72.037453830" watchObservedRunningTime="2026-09-17 23:12:18.994033068 +0000 UTC m=+72.038409227"
Sep 17 23:12:19.008090 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:19.008050    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8gs2j" podStartSLOduration=66.738504931 podStartE2EDuration="1m11.008037909s" totalImagesPullingTime="4.269532978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:19.007562091 +0000 UTC m=+72.051938251" watchObservedRunningTime="2026-09-17 23:12:19.008037909 +0000 UTC m=+72.052414066"
Sep 17 23:12:45.256494 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:45.256374    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-s6b9s"]
Sep 17 23:12:45.260517 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:45.260498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-s6b9s"
Sep 17 23:12:45.263220 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:45.263202    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-h7pzg\""
Sep 17 23:12:45.270176 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:45.270155    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-s6b9s"]
Sep 17 23:12:45.409232 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:45.409198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhqbf\" (UniqueName: \"kubernetes.io/projected/cac347e3-b53f-4871-a2c2-4c536a2fa04c-kube-api-access-dhqbf\") pod \"network-check-source-7cbc45fddc-s6b9s\" (UID: \"cac347e3-b53f-4871-a2c2-4c536a2fa04c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-s6b9s"
Sep 17 23:12:45.510524 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:45.510439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhqbf\" (UniqueName: \"kubernetes.io/projected/cac347e3-b53f-4871-a2c2-4c536a2fa04c-kube-api-access-dhqbf\") pod \"network-check-source-7cbc45fddc-s6b9s\" (UID: \"cac347e3-b53f-4871-a2c2-4c536a2fa04c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-s6b9s"
Sep 17 23:12:45.519223 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:45.519196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhqbf\" (UniqueName: \"kubernetes.io/projected/cac347e3-b53f-4871-a2c2-4c536a2fa04c-kube-api-access-dhqbf\") pod \"network-check-source-7cbc45fddc-s6b9s\" (UID: \"cac347e3-b53f-4871-a2c2-4c536a2fa04c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-s6b9s"
Sep 17 23:12:45.568555 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:45.568514    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-s6b9s"
Sep 17 23:12:45.712353 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:45.712312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls\") pod \"image-registry-7986d6c7dd-sk4nw\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") " pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:12:45.712527 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:45.712444    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:45.712527 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:45.712459    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7986d6c7dd-sk4nw: secret "image-registry-tls" not found
Sep 17 23:12:45.712527 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:45.712523    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls podName:ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a nodeName:}" failed. No retries permitted until 2026-09-17 23:13:49.712507095 +0000 UTC m=+162.756883253 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls") pod "image-registry-7986d6c7dd-sk4nw" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a") : secret "image-registry-tls" not found
Sep 17 23:12:45.813192 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:45.813154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:12:45.813342 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:45.813320    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:12:45.813398 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:45.813387    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls podName:6be6dc27-bc71-4d36-a57f-7f3b8557d0c3 nodeName:}" failed. No retries permitted until 2026-09-17 23:13:49.813372349 +0000 UTC m=+162.857748487 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls") pod "dns-default-x8v6t" (UID: "6be6dc27-bc71-4d36-a57f-7f3b8557d0c3") : secret "dns-default-metrics-tls" not found
Sep 17 23:12:45.846019 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:45.845987    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-s6b9s"]
Sep 17 23:12:45.849581 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:12:45.849554    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac347e3_b53f_4871_a2c2_4c536a2fa04c.slice/crio-8832feaace253f44294c2927340b5f57dbd65ae2f8f0e048c02cdc92b1c994a7 WatchSource:0}: Error finding container 8832feaace253f44294c2927340b5f57dbd65ae2f8f0e048c02cdc92b1c994a7: Status 404 returned error can't find the container with id 8832feaace253f44294c2927340b5f57dbd65ae2f8f0e048c02cdc92b1c994a7
Sep 17 23:12:46.061125 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:46.061082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-s6b9s" event={"ID":"cac347e3-b53f-4871-a2c2-4c536a2fa04c","Type":"ContainerStarted","Data":"443e4344f470f43e5c739ee0d878f74149bca588e889aa8dd66659892dde73b0"}
Sep 17 23:12:46.061309 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:46.061139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-s6b9s" event={"ID":"cac347e3-b53f-4871-a2c2-4c536a2fa04c","Type":"ContainerStarted","Data":"8832feaace253f44294c2927340b5f57dbd65ae2f8f0e048c02cdc92b1c994a7"}
Sep 17 23:12:46.076789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:46.076694    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-s6b9s" podStartSLOduration=1.076678733 podStartE2EDuration="1.076678733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:46.07624167 +0000 UTC m=+99.120617831" watchObservedRunningTime="2026-09-17 23:12:46.076678733 +0000 UTC m=+99.121054893"
Sep 17 23:12:49.978907 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:49.978877    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-72k7x"]
Sep 17 23:12:50.010780 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.010757    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8gs2j"
Sep 17 23:12:50.010928 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.010788    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-72k7x"]
Sep 17 23:12:50.010977 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.010949    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.016308 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.016287    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 23:12:50.017224 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.017206    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 23:12:50.017307 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.017225    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 23:12:50.018898 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.018876    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8qgl2\""
Sep 17 23:12:50.032875 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.032859    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 23:12:50.143206 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.143170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/924de91b-c17e-4d3b-933c-f8129c8b3265-data-volume\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.143206 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.143206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.143430 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.143327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxfxt\" (UniqueName: \"kubernetes.io/projected/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-api-access-qxfxt\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.143465 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.143453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/924de91b-c17e-4d3b-933c-f8129c8b3265-crio-socket\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.143502 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.143488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.244348 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.244260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/924de91b-c17e-4d3b-933c-f8129c8b3265-data-volume\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.244348 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.244296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.244348 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.244318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxfxt\" (UniqueName: \"kubernetes.io/projected/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-api-access-qxfxt\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.244348 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.244345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/924de91b-c17e-4d3b-933c-f8129c8b3265-crio-socket\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.244758 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.244384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.244758 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:50.244394    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:12:50.244758 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:50.244475    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:50.244758 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:50.244487    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm podName:924de91b-c17e-4d3b-933c-f8129c8b3265 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:50.744459415 +0000 UTC m=+103.788835570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72k7x" (UID: "924de91b-c17e-4d3b-933c-f8129c8b3265") : configmap "kube-rbac-proxy" not found
Sep 17 23:12:50.244758 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.244501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/924de91b-c17e-4d3b-933c-f8129c8b3265-crio-socket\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.244758 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:50.244530    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls podName:924de91b-c17e-4d3b-933c-f8129c8b3265 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:50.744514201 +0000 UTC m=+103.788890340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72k7x" (UID: "924de91b-c17e-4d3b-933c-f8129c8b3265") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:50.244758 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.244703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/924de91b-c17e-4d3b-933c-f8129c8b3265-data-volume\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.258864 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.258839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxfxt\" (UniqueName: \"kubernetes.io/projected/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-api-access-qxfxt\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.748958 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.748921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.749134 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:50.748993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:50.749134 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:50.749052    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:12:50.749134 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:50.749097    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:50.749134 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:50.749126    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm podName:924de91b-c17e-4d3b-933c-f8129c8b3265 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:51.749109513 +0000 UTC m=+104.793485654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72k7x" (UID: "924de91b-c17e-4d3b-933c-f8129c8b3265") : configmap "kube-rbac-proxy" not found
Sep 17 23:12:50.749280 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:50.749151    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls podName:924de91b-c17e-4d3b-933c-f8129c8b3265 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:51.749137973 +0000 UTC m=+104.793514115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72k7x" (UID: "924de91b-c17e-4d3b-933c-f8129c8b3265") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:51.756698 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:51.756659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:51.757163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:51.756727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:51.757163 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:51.756783    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:12:51.757163 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:51.756806    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:51.757163 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:51.756832    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm podName:924de91b-c17e-4d3b-933c-f8129c8b3265 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:53.7568199 +0000 UTC m=+106.801196039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72k7x" (UID: "924de91b-c17e-4d3b-933c-f8129c8b3265") : configmap "kube-rbac-proxy" not found
Sep 17 23:12:51.757163 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:51.756864    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls podName:924de91b-c17e-4d3b-933c-f8129c8b3265 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:53.756851167 +0000 UTC m=+106.801227306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72k7x" (UID: "924de91b-c17e-4d3b-933c-f8129c8b3265") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:51.942991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:51.942957    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t65zf_60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5/dns-node-resolver/0.log"
Sep 17 23:12:52.948228 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:52.948193    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kcvpd_265e74c8-a31a-43de-af18-711758cc8e5d/node-ca/0.log"
Sep 17 23:12:53.771860 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:53.771824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:53.772028 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:53.771881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:53.772028 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:53.771942    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:12:53.772028 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:53.771976    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:53.772028 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:53.772014    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm podName:924de91b-c17e-4d3b-933c-f8129c8b3265 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:57.771999336 +0000 UTC m=+110.816375474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72k7x" (UID: "924de91b-c17e-4d3b-933c-f8129c8b3265") : configmap "kube-rbac-proxy" not found
Sep 17 23:12:53.772028 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:53.772029    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls podName:924de91b-c17e-4d3b-933c-f8129c8b3265 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:57.772023182 +0000 UTC m=+110.816399320 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72k7x" (UID: "924de91b-c17e-4d3b-933c-f8129c8b3265") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:53.942904 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:53.942876    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rj6f4_11ed4035-5836-4a4a-ab0e-d73776ada4cf/serve-healthcheck-canary/0.log"
Sep 17 23:12:57.802743 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:57.802701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:57.803131 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:12:57.802763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:12:57.803131 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:57.802819    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:12:57.803131 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:57.802845    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:57.803131 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:57.802880    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm podName:924de91b-c17e-4d3b-933c-f8129c8b3265 nodeName:}" failed. No retries permitted until 2026-09-17 23:13:05.802866741 +0000 UTC m=+118.847242880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72k7x" (UID: "924de91b-c17e-4d3b-933c-f8129c8b3265") : configmap "kube-rbac-proxy" not found
Sep 17 23:12:57.803131 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:12:57.802894    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls podName:924de91b-c17e-4d3b-933c-f8129c8b3265 nodeName:}" failed. No retries permitted until 2026-09-17 23:13:05.802888257 +0000 UTC m=+118.847264395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72k7x" (UID: "924de91b-c17e-4d3b-933c-f8129c8b3265") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:13:05.865589 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:05.865512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:13:05.865996 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:05.865645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:13:05.866140 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:05.866121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/924de91b-c17e-4d3b-933c-f8129c8b3265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:13:05.867977 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:05.867943    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/924de91b-c17e-4d3b-933c-f8129c8b3265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72k7x\" (UID: \"924de91b-c17e-4d3b-933c-f8129c8b3265\") " pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:13:05.919336 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:05.919305    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-72k7x"
Sep 17 23:13:06.186847 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:06.186772    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-72k7x"]
Sep 17 23:13:06.189584 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:13:06.189557    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924de91b_c17e_4d3b_933c_f8129c8b3265.slice/crio-44d871920a2be3db6d6d6ee9a3537047e5f47dea8b7a9d4c83d3eea9219c70a2 WatchSource:0}: Error finding container 44d871920a2be3db6d6d6ee9a3537047e5f47dea8b7a9d4c83d3eea9219c70a2: Status 404 returned error can't find the container with id 44d871920a2be3db6d6d6ee9a3537047e5f47dea8b7a9d4c83d3eea9219c70a2
Sep 17 23:13:07.124687 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:07.124581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-72k7x" event={"ID":"924de91b-c17e-4d3b-933c-f8129c8b3265","Type":"ContainerStarted","Data":"d633e510da70b0a36b9c945eb5b04814d63e894d4a54f33b803931376f52200b"}
Sep 17 23:13:07.124687 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:07.124655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-72k7x" event={"ID":"924de91b-c17e-4d3b-933c-f8129c8b3265","Type":"ContainerStarted","Data":"c0886ce1302f30fb912ebe56bd3b8aede098a1683e8512829712c9b23bf6173e"}
Sep 17 23:13:07.124687 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:07.124673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-72k7x" event={"ID":"924de91b-c17e-4d3b-933c-f8129c8b3265","Type":"ContainerStarted","Data":"44d871920a2be3db6d6d6ee9a3537047e5f47dea8b7a9d4c83d3eea9219c70a2"}
Sep 17 23:13:09.133154 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:09.133115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-72k7x" event={"ID":"924de91b-c17e-4d3b-933c-f8129c8b3265","Type":"ContainerStarted","Data":"d7633b47902af6ec93af0457cc163e8c484169860f5b0bfe13b9dea6f57a73e5"}
Sep 17 23:13:09.151724 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:09.151679    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-72k7x" podStartSLOduration=18.268210092 podStartE2EDuration="20.151661326s" totalImagesPullingTime="1.883451234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:12:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:09.151041774 +0000 UTC m=+122.195417933" watchObservedRunningTime="2026-09-17 23:13:09.151661326 +0000 UTC m=+122.196037486"
Sep 17 23:13:12.196953 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:12.196910    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw" podUID="3f113e0e-9447-42c5-9893-d2bbca7c4403" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 23:13:17.252301 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:17.252248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:13:17.254723 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:17.254702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e6da33b-9d1e-472a-97aa-e4951d3972ff-metrics-certs\") pod \"network-metrics-daemon-gr7p6\" (UID: \"1e6da33b-9d1e-472a-97aa-e4951d3972ff\") " pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:13:17.426705 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:17.426676    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zzvsp\""
Sep 17 23:13:17.433945 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:17.433920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7p6"
Sep 17 23:13:17.703940 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:17.703908    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gr7p6"]
Sep 17 23:13:17.707922 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:13:17.707898    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6da33b_9d1e_472a_97aa_e4951d3972ff.slice/crio-f94137b51047bafde437844e79f025e5347c293a2e55b87f78fe6c9bec1008da WatchSource:0}: Error finding container f94137b51047bafde437844e79f025e5347c293a2e55b87f78fe6c9bec1008da: Status 404 returned error can't find the container with id f94137b51047bafde437844e79f025e5347c293a2e55b87f78fe6c9bec1008da
Sep 17 23:13:18.161131 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:18.161094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gr7p6" event={"ID":"1e6da33b-9d1e-472a-97aa-e4951d3972ff","Type":"ContainerStarted","Data":"f94137b51047bafde437844e79f025e5347c293a2e55b87f78fe6c9bec1008da"}
Sep 17 23:13:19.165490 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:19.165456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gr7p6" event={"ID":"1e6da33b-9d1e-472a-97aa-e4951d3972ff","Type":"ContainerStarted","Data":"3c555db1b063949b4e7b2c58401b975883e00d14a04a100211040d6cda9c6889"}
Sep 17 23:13:19.165900 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:19.165535    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gr7p6" event={"ID":"1e6da33b-9d1e-472a-97aa-e4951d3972ff","Type":"ContainerStarted","Data":"2c27829b3c484fe525cb7f730ffceabef8b83f3a839c2391a3e7f99f0acc960b"}
Sep 17 23:13:19.184498 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:19.184453    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gr7p6" podStartSLOduration=130.125785105 podStartE2EDuration="2m11.184439413s" totalImagesPullingTime="1.058654308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:19.18382436 +0000 UTC m=+132.228200520" watchObservedRunningTime="2026-09-17 23:13:19.184439413 +0000 UTC m=+132.228815572"
Sep 17 23:13:22.196451 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:22.196416    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw" podUID="3f113e0e-9447-42c5-9893-d2bbca7c4403" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 23:13:27.222755 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.222722    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-csnjv"]
Sep 17 23:13:27.225809 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.225793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.228149 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.228126    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 23:13:27.228279 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.228179    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 23:13:27.229074 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.229059    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4h2zn\""
Sep 17 23:13:27.229295 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.229278    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 23:13:27.229363 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.229332    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 23:13:27.229363 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.229353    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 23:13:27.229444 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.229373    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 23:13:27.332218 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.332190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1913439d-3dd6-448f-8e13-31fecc7456a2-root\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.332218 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.332220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1913439d-3dd6-448f-8e13-31fecc7456a2-metrics-client-ca\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.332380 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.332243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.332380 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.332330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-wtmp\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.332380 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.332362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-textfile\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.332475 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.332386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmf4\" (UniqueName: \"kubernetes.io/projected/1913439d-3dd6-448f-8e13-31fecc7456a2-kube-api-access-ltmf4\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.332475 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.332405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-tls\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.332475 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.332430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1913439d-3dd6-448f-8e13-31fecc7456a2-sys\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.332565 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.332481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.433685 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.433652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-wtmp\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.433685 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.433687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-textfile\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.433883 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.433709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmf4\" (UniqueName: \"kubernetes.io/projected/1913439d-3dd6-448f-8e13-31fecc7456a2-kube-api-access-ltmf4\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.433883 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.433737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-tls\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.433883 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.433761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1913439d-3dd6-448f-8e13-31fecc7456a2-sys\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.433883 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.433788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.433883 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.433830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1913439d-3dd6-448f-8e13-31fecc7456a2-root\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.433883 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.433844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-wtmp\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.433883 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:13:27.433850    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 23:13:27.434208 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.433900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1913439d-3dd6-448f-8e13-31fecc7456a2-sys\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.434208 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:13:27.433914    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-tls podName:1913439d-3dd6-448f-8e13-31fecc7456a2 nodeName:}" failed. No retries permitted until 2026-09-17 23:13:27.933899018 +0000 UTC m=+140.978275156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-tls") pod "node-exporter-csnjv" (UID: "1913439d-3dd6-448f-8e13-31fecc7456a2") : secret "node-exporter-tls" not found
Sep 17 23:13:27.434208 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.433906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1913439d-3dd6-448f-8e13-31fecc7456a2-root\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.434208 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.433852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1913439d-3dd6-448f-8e13-31fecc7456a2-metrics-client-ca\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.434208 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.433995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.434208 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.434133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-textfile\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.434466 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.434344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.434466 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.434389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1913439d-3dd6-448f-8e13-31fecc7456a2-metrics-client-ca\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.436418 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.436385    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.442900 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.442881    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmf4\" (UniqueName: \"kubernetes.io/projected/1913439d-3dd6-448f-8e13-31fecc7456a2-kube-api-access-ltmf4\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.938157 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:27.938125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-tls\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:27.938343 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:13:27.938263    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 23:13:27.938343 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:13:27.938317    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-tls podName:1913439d-3dd6-448f-8e13-31fecc7456a2 nodeName:}" failed. No retries permitted until 2026-09-17 23:13:28.938304687 +0000 UTC m=+141.982680825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-tls") pod "node-exporter-csnjv" (UID: "1913439d-3dd6-448f-8e13-31fecc7456a2") : secret "node-exporter-tls" not found
Sep 17 23:13:28.946516 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:28.946476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-tls\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:28.948886 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:28.948868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1913439d-3dd6-448f-8e13-31fecc7456a2-node-exporter-tls\") pod \"node-exporter-csnjv\" (UID: \"1913439d-3dd6-448f-8e13-31fecc7456a2\") " pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:29.035276 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:29.035225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-csnjv"
Sep 17 23:13:29.045724 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:13:29.045694    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1913439d_3dd6_448f_8e13_31fecc7456a2.slice/crio-5517de2a24dbeac649987d8e635ca8e0a95c14e272a424f95a1b977e8bba1777 WatchSource:0}: Error finding container 5517de2a24dbeac649987d8e635ca8e0a95c14e272a424f95a1b977e8bba1777: Status 404 returned error can't find the container with id 5517de2a24dbeac649987d8e635ca8e0a95c14e272a424f95a1b977e8bba1777
Sep 17 23:13:29.201178 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:29.201117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-csnjv" event={"ID":"1913439d-3dd6-448f-8e13-31fecc7456a2","Type":"ContainerStarted","Data":"5517de2a24dbeac649987d8e635ca8e0a95c14e272a424f95a1b977e8bba1777"}
Sep 17 23:13:30.205784 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:30.205743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-csnjv" event={"ID":"1913439d-3dd6-448f-8e13-31fecc7456a2","Type":"ContainerStarted","Data":"7d908637894ddd6ef4489fc03b3801032a81f9db04d9333fc6747e08ce75ab30"}
Sep 17 23:13:31.210404 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:31.210371    2580 generic.go:356] "Generic (PLEG): container finished" podID="1913439d-3dd6-448f-8e13-31fecc7456a2" containerID="7d908637894ddd6ef4489fc03b3801032a81f9db04d9333fc6747e08ce75ab30" exitCode=0
Sep 17 23:13:31.210845 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:31.210457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-csnjv" event={"ID":"1913439d-3dd6-448f-8e13-31fecc7456a2","Type":"ContainerDied","Data":"7d908637894ddd6ef4489fc03b3801032a81f9db04d9333fc6747e08ce75ab30"}
Sep 17 23:13:32.196701 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:32.196661    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw" podUID="3f113e0e-9447-42c5-9893-d2bbca7c4403" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 23:13:32.196867 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:32.196740    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw"
Sep 17 23:13:32.197259 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:32.197228    2580 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"20e18a00541fa08a872c703c542e80de16fdf364f4408d22a3394f8f041d8566"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 17 23:13:32.197300 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:32.197283    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw" podUID="3f113e0e-9447-42c5-9893-d2bbca7c4403" containerName="service-proxy" containerID="cri-o://20e18a00541fa08a872c703c542e80de16fdf364f4408d22a3394f8f041d8566" gracePeriod=30
Sep 17 23:13:32.215537 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:32.215513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-csnjv" event={"ID":"1913439d-3dd6-448f-8e13-31fecc7456a2","Type":"ContainerStarted","Data":"86ccc22b256eff58c3098759494fb1ed72b4d56657caaff3533137afd1a4edc1"}
Sep 17 23:13:32.215842 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:32.215560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-csnjv" event={"ID":"1913439d-3dd6-448f-8e13-31fecc7456a2","Type":"ContainerStarted","Data":"6d37eb6790aee46317a397e8c88311dd364dac944cd615be9e4a988626da149f"}
Sep 17 23:13:32.235534 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:32.235492    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-csnjv" podStartSLOduration=4.145019249 podStartE2EDuration="5.235478533s" totalImagesPullingTime="1.022345132s" totalInitContainerRuntime="68.114152ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:32.235238096 +0000 UTC m=+145.279614257" watchObservedRunningTime="2026-09-17 23:13:32.235478533 +0000 UTC m=+145.279854692"
Sep 17 23:13:33.221324 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:33.221292    2580 generic.go:356] "Generic (PLEG): container finished" podID="3f113e0e-9447-42c5-9893-d2bbca7c4403" containerID="20e18a00541fa08a872c703c542e80de16fdf364f4408d22a3394f8f041d8566" exitCode=2
Sep 17 23:13:33.221721 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:33.221379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw" event={"ID":"3f113e0e-9447-42c5-9893-d2bbca7c4403","Type":"ContainerDied","Data":"20e18a00541fa08a872c703c542e80de16fdf364f4408d22a3394f8f041d8566"}
Sep 17 23:13:33.221721 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:33.221432    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65dffc8776-6fngw" event={"ID":"3f113e0e-9447-42c5-9893-d2bbca7c4403","Type":"ContainerStarted","Data":"7bb3a7cb80159b269fc951683ad9d77c58b12c1ca118e0269c9b6d775edecdba"}
Sep 17 23:13:36.717404 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:36.717370    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7986d6c7dd-sk4nw"]
Sep 17 23:13:36.717775 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:13:36.717546    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw" podUID="ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a"
Sep 17 23:13:37.234811 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.234779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:13:37.241031 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.241002    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:13:37.305721 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.305689    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-ca-trust-extracted\") pod \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") "
Sep 17 23:13:37.305877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.305729    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-certificates\") pod \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") "
Sep 17 23:13:37.305877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.305763    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-bound-sa-token\") pod \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") "
Sep 17 23:13:37.305877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.305797    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrdhb\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-kube-api-access-xrdhb\") pod \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") "
Sep 17 23:13:37.305877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.305834    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-installation-pull-secrets\") pod \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") "
Sep 17 23:13:37.305877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.305867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-trusted-ca\") pod \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") "
Sep 17 23:13:37.306103 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.305903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-image-registry-private-configuration\") pod \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\" (UID: \"ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a\") "
Sep 17 23:13:37.306103 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.305997    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:13:37.306203 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.306101    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:13:37.306203 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.306126    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-ca-trust-extracted\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:13:37.306416 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.306378    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:13:37.308405 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.308376    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:13:37.308520 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.308447    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:37.308679 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.308652    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:13:37.308781 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.308746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-kube-api-access-xrdhb" (OuterVolumeSpecName: "kube-api-access-xrdhb") pod "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a" (UID: "ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a"). InnerVolumeSpecName "kube-api-access-xrdhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:37.406887 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.406857    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-image-registry-private-configuration\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:13:37.406887 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.406881    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-certificates\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:13:37.406887 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.406892    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-bound-sa-token\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:13:37.407075 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.406901    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrdhb\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-kube-api-access-xrdhb\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:13:37.407075 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.406910    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-installation-pull-secrets\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:13:37.407075 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:37.406919    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-trusted-ca\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:13:38.237475 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:38.237446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7986d6c7dd-sk4nw"
Sep 17 23:13:38.287627 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:38.287592    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7986d6c7dd-sk4nw"]
Sep 17 23:13:38.297211 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:38.297179    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7986d6c7dd-sk4nw"]
Sep 17 23:13:38.413719 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:38.413690    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a-registry-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:13:39.501400 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:39.501370    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a" path="/var/lib/kubelet/pods/ecc8f2a2-efa5-433c-bd84-f4a4c0c01b0a/volumes"
Sep 17 23:13:44.914238 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:13:44.914206    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-x8v6t" podUID="6be6dc27-bc71-4d36-a57f-7f3b8557d0c3"
Sep 17 23:13:45.258235 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:45.258161    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:13:49.900114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:49.900084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:13:49.902507 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:49.902474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be6dc27-bc71-4d36-a57f-7f3b8557d0c3-metrics-tls\") pod \"dns-default-x8v6t\" (UID: \"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3\") " pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:13:50.060862 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:50.060830    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9p4tt\""
Sep 17 23:13:50.068755 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:50.068732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:13:50.341425 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:50.341322    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x8v6t"]
Sep 17 23:13:50.343873 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:13:50.343842    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be6dc27_bc71_4d36_a57f_7f3b8557d0c3.slice/crio-fbbaf4ba7bf9cd00857874c0f6ed91efd3f820782b17bd492d12f3e1cf1b711d WatchSource:0}: Error finding container fbbaf4ba7bf9cd00857874c0f6ed91efd3f820782b17bd492d12f3e1cf1b711d: Status 404 returned error can't find the container with id fbbaf4ba7bf9cd00857874c0f6ed91efd3f820782b17bd492d12f3e1cf1b711d
Sep 17 23:13:51.279923 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:51.279880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x8v6t" event={"ID":"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3","Type":"ContainerStarted","Data":"fbbaf4ba7bf9cd00857874c0f6ed91efd3f820782b17bd492d12f3e1cf1b711d"}
Sep 17 23:13:52.285468 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:52.285438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x8v6t" event={"ID":"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3","Type":"ContainerStarted","Data":"579667b449445b98e2606a676ff0ea43ce8e1e34d018f7b0741511ae58f13718"}
Sep 17 23:13:52.285830 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:52.285490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x8v6t" event={"ID":"6be6dc27-bc71-4d36-a57f-7f3b8557d0c3","Type":"ContainerStarted","Data":"057cbdbdd987db964ee6c28c842534d573e10454fef13fd5b4c9704a142c3d7e"}
Sep 17 23:13:52.285830 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:52.285569    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:13:52.304628 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:13:52.304581    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x8v6t" podStartSLOduration=129.583873169 podStartE2EDuration="2m11.304565894s" totalImagesPullingTime="1.720692725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:52.303603367 +0000 UTC m=+165.347979540" watchObservedRunningTime="2026-09-17 23:13:52.304565894 +0000 UTC m=+165.348942055"
Sep 17 23:14:02.291262 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:14:02.291179    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x8v6t"
Sep 17 23:16:07.382676 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:07.382651    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 23:16:32.770671 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.770626    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nxxcl"]
Sep 17 23:16:32.773921 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.773905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-nxxcl"
Sep 17 23:16:32.776341 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.776313    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 23:16:32.776459 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.776364    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-x4pxk\""
Sep 17 23:16:32.776459 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.776372    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 23:16:32.777357 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.777335    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 23:16:32.777465 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.777392    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 23:16:32.780776 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.780443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a17612b0-68f7-41b0-9589-65dc2edfd08b-certificates\") pod \"keda-admission-6c4d864b86-nxxcl\" (UID: \"a17612b0-68f7-41b0-9589-65dc2edfd08b\") " pod="openshift-keda/keda-admission-6c4d864b86-nxxcl"
Sep 17 23:16:32.780776 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.780488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6hdm\" (UniqueName: \"kubernetes.io/projected/a17612b0-68f7-41b0-9589-65dc2edfd08b-kube-api-access-x6hdm\") pod \"keda-admission-6c4d864b86-nxxcl\" (UID: \"a17612b0-68f7-41b0-9589-65dc2edfd08b\") " pod="openshift-keda/keda-admission-6c4d864b86-nxxcl"
Sep 17 23:16:32.783243 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.783215    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nxxcl"]
Sep 17 23:16:32.881938 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.881912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a17612b0-68f7-41b0-9589-65dc2edfd08b-certificates\") pod \"keda-admission-6c4d864b86-nxxcl\" (UID: \"a17612b0-68f7-41b0-9589-65dc2edfd08b\") " pod="openshift-keda/keda-admission-6c4d864b86-nxxcl"
Sep 17 23:16:32.882114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.881946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6hdm\" (UniqueName: \"kubernetes.io/projected/a17612b0-68f7-41b0-9589-65dc2edfd08b-kube-api-access-x6hdm\") pod \"keda-admission-6c4d864b86-nxxcl\" (UID: \"a17612b0-68f7-41b0-9589-65dc2edfd08b\") " pod="openshift-keda/keda-admission-6c4d864b86-nxxcl"
Sep 17 23:16:32.885968 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:16:32.882354    2580 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 23:16:32.885968 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:16:32.882385    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-nxxcl: secret "keda-admission-webhooks-certs" not found
Sep 17 23:16:32.885968 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:16:32.882463    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a17612b0-68f7-41b0-9589-65dc2edfd08b-certificates podName:a17612b0-68f7-41b0-9589-65dc2edfd08b nodeName:}" failed. No retries permitted until 2026-09-17 23:16:33.382442247 +0000 UTC m=+326.426818388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a17612b0-68f7-41b0-9589-65dc2edfd08b-certificates") pod "keda-admission-6c4d864b86-nxxcl" (UID: "a17612b0-68f7-41b0-9589-65dc2edfd08b") : secret "keda-admission-webhooks-certs" not found
Sep 17 23:16:32.891011 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:32.890986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6hdm\" (UniqueName: \"kubernetes.io/projected/a17612b0-68f7-41b0-9589-65dc2edfd08b-kube-api-access-x6hdm\") pod \"keda-admission-6c4d864b86-nxxcl\" (UID: \"a17612b0-68f7-41b0-9589-65dc2edfd08b\") " pod="openshift-keda/keda-admission-6c4d864b86-nxxcl"
Sep 17 23:16:33.385276 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:33.385240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a17612b0-68f7-41b0-9589-65dc2edfd08b-certificates\") pod \"keda-admission-6c4d864b86-nxxcl\" (UID: \"a17612b0-68f7-41b0-9589-65dc2edfd08b\") " pod="openshift-keda/keda-admission-6c4d864b86-nxxcl"
Sep 17 23:16:33.387866 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:33.387841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a17612b0-68f7-41b0-9589-65dc2edfd08b-certificates\") pod \"keda-admission-6c4d864b86-nxxcl\" (UID: \"a17612b0-68f7-41b0-9589-65dc2edfd08b\") " pod="openshift-keda/keda-admission-6c4d864b86-nxxcl"
Sep 17 23:16:33.684998 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:33.684914    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-nxxcl"
Sep 17 23:16:33.952969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:33.952879    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nxxcl"]
Sep 17 23:16:33.956528 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:16:33.956504    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17612b0_68f7_41b0_9589_65dc2edfd08b.slice/crio-77980cdeab1dade5cc49c98eff8c1b3a755503d1da37773a28bedb3f98d4eba2 WatchSource:0}: Error finding container 77980cdeab1dade5cc49c98eff8c1b3a755503d1da37773a28bedb3f98d4eba2: Status 404 returned error can't find the container with id 77980cdeab1dade5cc49c98eff8c1b3a755503d1da37773a28bedb3f98d4eba2
Sep 17 23:16:33.958252 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:33.958237    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:16:34.816485 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:34.816446    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-nxxcl" event={"ID":"a17612b0-68f7-41b0-9589-65dc2edfd08b","Type":"ContainerStarted","Data":"77980cdeab1dade5cc49c98eff8c1b3a755503d1da37773a28bedb3f98d4eba2"}
Sep 17 23:16:36.824790 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:36.824760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-nxxcl" event={"ID":"a17612b0-68f7-41b0-9589-65dc2edfd08b","Type":"ContainerStarted","Data":"13687bd9b980f3d069d6cbcc7e76549fbf0fcd5cbbf68de53b9d69f33f11a52a"}
Sep 17 23:16:36.825148 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:36.824863    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-nxxcl"
Sep 17 23:16:36.842093 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:36.842055    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-nxxcl" podStartSLOduration=2.365571182 podStartE2EDuration="4.842041891s" totalImagesPullingTime="2.476470709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:36.841007914 +0000 UTC m=+329.885384073" watchObservedRunningTime="2026-09-17 23:16:36.842041891 +0000 UTC m=+329.886418050"
Sep 17 23:16:57.830347 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:16:57.830317    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-nxxcl"
Sep 17 23:17:45.718215 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.718135    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"]
Sep 17 23:17:45.721263 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.721248    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"
Sep 17 23:17:45.723360 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.723339    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-2b2wj\""
Sep 17 23:17:45.723479 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.723344    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 23:17:45.724309 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.724290    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 23:17:45.724402 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.724308    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 23:17:45.731834 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.731811    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"]
Sep 17 23:17:45.767342 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.767307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shrbl\" (UniqueName: \"kubernetes.io/projected/50ae5543-eb37-4b06-8384-f7db8e0df56a-kube-api-access-shrbl\") pod \"llmisvc-controller-manager-645f9c8fd6-rqr2q\" (UID: \"50ae5543-eb37-4b06-8384-f7db8e0df56a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"
Sep 17 23:17:45.767493 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.767367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50ae5543-eb37-4b06-8384-f7db8e0df56a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-rqr2q\" (UID: \"50ae5543-eb37-4b06-8384-f7db8e0df56a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"
Sep 17 23:17:45.867907 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.867874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50ae5543-eb37-4b06-8384-f7db8e0df56a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-rqr2q\" (UID: \"50ae5543-eb37-4b06-8384-f7db8e0df56a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"
Sep 17 23:17:45.868088 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.867922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shrbl\" (UniqueName: \"kubernetes.io/projected/50ae5543-eb37-4b06-8384-f7db8e0df56a-kube-api-access-shrbl\") pod \"llmisvc-controller-manager-645f9c8fd6-rqr2q\" (UID: \"50ae5543-eb37-4b06-8384-f7db8e0df56a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"
Sep 17 23:17:45.870572 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.870537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50ae5543-eb37-4b06-8384-f7db8e0df56a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-rqr2q\" (UID: \"50ae5543-eb37-4b06-8384-f7db8e0df56a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"
Sep 17 23:17:45.876727 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:45.876706    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shrbl\" (UniqueName: \"kubernetes.io/projected/50ae5543-eb37-4b06-8384-f7db8e0df56a-kube-api-access-shrbl\") pod \"llmisvc-controller-manager-645f9c8fd6-rqr2q\" (UID: \"50ae5543-eb37-4b06-8384-f7db8e0df56a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"
Sep 17 23:17:46.030078 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:46.029987    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"
Sep 17 23:17:46.298709 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:46.298676    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"]
Sep 17 23:17:46.303325 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:17:46.303299    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod50ae5543_eb37_4b06_8384_f7db8e0df56a.slice/crio-c43ef6361fd8dd3cbd516985a3472977145127a8e83de5d35d421f6ecc8d0db3 WatchSource:0}: Error finding container c43ef6361fd8dd3cbd516985a3472977145127a8e83de5d35d421f6ecc8d0db3: Status 404 returned error can't find the container with id c43ef6361fd8dd3cbd516985a3472977145127a8e83de5d35d421f6ecc8d0db3
Sep 17 23:17:47.062676 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:47.062632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q" event={"ID":"50ae5543-eb37-4b06-8384-f7db8e0df56a","Type":"ContainerStarted","Data":"c43ef6361fd8dd3cbd516985a3472977145127a8e83de5d35d421f6ecc8d0db3"}
Sep 17 23:17:49.071392 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:49.071357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q" event={"ID":"50ae5543-eb37-4b06-8384-f7db8e0df56a","Type":"ContainerStarted","Data":"630448f46645ef980c73e84425a9ee16afbc3e8552362ae86ebbc08fef98e958"}
Sep 17 23:17:49.071825 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:49.071446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"
Sep 17 23:17:49.088101 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:17:49.088052    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q" podStartSLOduration=1.814069721 podStartE2EDuration="4.088036588s" totalImagesPullingTime="2.273966867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:49.086179747 +0000 UTC m=+402.130555907" watchObservedRunningTime="2026-09-17 23:17:49.088036588 +0000 UTC m=+402.132412747"
Sep 17 23:18:20.077574 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:18:20.077542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-rqr2q"
Sep 17 23:19:00.853253 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:00.853225    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-8smpb"]
Sep 17 23:19:00.857602 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:00.857582    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:00.859929 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:00.859906    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-p4wwh\""
Sep 17 23:19:00.860055 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:00.859949    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 23:19:00.860055 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:00.859949    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 23:19:00.865108 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:00.865085    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-8smpb"]
Sep 17 23:19:00.900223 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:00.900180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86tk8\" (UniqueName: \"kubernetes.io/projected/0137c215-8ab6-4514-bf18-b8bd1934d8d3-kube-api-access-86tk8\") pod \"odh-model-controller-7b966d7984-8smpb\" (UID: \"0137c215-8ab6-4514-bf18-b8bd1934d8d3\") " pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:00.900400 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:00.900236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0137c215-8ab6-4514-bf18-b8bd1934d8d3-cert\") pod \"odh-model-controller-7b966d7984-8smpb\" (UID: \"0137c215-8ab6-4514-bf18-b8bd1934d8d3\") " pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:00.900400 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:00.900311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0137c215-8ab6-4514-bf18-b8bd1934d8d3-metrics-certs\") pod \"odh-model-controller-7b966d7984-8smpb\" (UID: \"0137c215-8ab6-4514-bf18-b8bd1934d8d3\") " pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:01.001469 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:01.001435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0137c215-8ab6-4514-bf18-b8bd1934d8d3-cert\") pod \"odh-model-controller-7b966d7984-8smpb\" (UID: \"0137c215-8ab6-4514-bf18-b8bd1934d8d3\") " pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:01.001654 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:01.001478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0137c215-8ab6-4514-bf18-b8bd1934d8d3-metrics-certs\") pod \"odh-model-controller-7b966d7984-8smpb\" (UID: \"0137c215-8ab6-4514-bf18-b8bd1934d8d3\") " pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:01.001654 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:01.001515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86tk8\" (UniqueName: \"kubernetes.io/projected/0137c215-8ab6-4514-bf18-b8bd1934d8d3-kube-api-access-86tk8\") pod \"odh-model-controller-7b966d7984-8smpb\" (UID: \"0137c215-8ab6-4514-bf18-b8bd1934d8d3\") " pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:01.001654 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:19:01.001584    2580 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 23:19:01.001759 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:19:01.001678    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0137c215-8ab6-4514-bf18-b8bd1934d8d3-cert podName:0137c215-8ab6-4514-bf18-b8bd1934d8d3 nodeName:}" failed. No retries permitted until 2026-09-17 23:19:01.501661693 +0000 UTC m=+474.546037833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0137c215-8ab6-4514-bf18-b8bd1934d8d3-cert") pod "odh-model-controller-7b966d7984-8smpb" (UID: "0137c215-8ab6-4514-bf18-b8bd1934d8d3") : secret "odh-model-controller-webhook-cert" not found
Sep 17 23:19:01.004067 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:01.004049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0137c215-8ab6-4514-bf18-b8bd1934d8d3-metrics-certs\") pod \"odh-model-controller-7b966d7984-8smpb\" (UID: \"0137c215-8ab6-4514-bf18-b8bd1934d8d3\") " pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:01.010701 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:01.010674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86tk8\" (UniqueName: \"kubernetes.io/projected/0137c215-8ab6-4514-bf18-b8bd1934d8d3-kube-api-access-86tk8\") pod \"odh-model-controller-7b966d7984-8smpb\" (UID: \"0137c215-8ab6-4514-bf18-b8bd1934d8d3\") " pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:01.505755 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:01.505720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0137c215-8ab6-4514-bf18-b8bd1934d8d3-cert\") pod \"odh-model-controller-7b966d7984-8smpb\" (UID: \"0137c215-8ab6-4514-bf18-b8bd1934d8d3\") " pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:01.508228 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:01.508204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0137c215-8ab6-4514-bf18-b8bd1934d8d3-cert\") pod \"odh-model-controller-7b966d7984-8smpb\" (UID: \"0137c215-8ab6-4514-bf18-b8bd1934d8d3\") " pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:01.768183 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:01.768071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:02.036339 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:02.036311    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-8smpb"]
Sep 17 23:19:02.039356 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:19:02.039332    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0137c215_8ab6_4514_bf18_b8bd1934d8d3.slice/crio-2d74d6a3503add21fb23b03adc9f0042c84d5c2b495c3dd830c33f56761dac98 WatchSource:0}: Error finding container 2d74d6a3503add21fb23b03adc9f0042c84d5c2b495c3dd830c33f56761dac98: Status 404 returned error can't find the container with id 2d74d6a3503add21fb23b03adc9f0042c84d5c2b495c3dd830c33f56761dac98
Sep 17 23:19:02.317827 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:02.317728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-8smpb" event={"ID":"0137c215-8ab6-4514-bf18-b8bd1934d8d3","Type":"ContainerStarted","Data":"2d74d6a3503add21fb23b03adc9f0042c84d5c2b495c3dd830c33f56761dac98"}
Sep 17 23:19:05.330198 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:05.330161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-8smpb" event={"ID":"0137c215-8ab6-4514-bf18-b8bd1934d8d3","Type":"ContainerStarted","Data":"bd3a9648c8de0799408ebec225a5aab3ff74f799c53eb134c63ae41a520fc2ec"}
Sep 17 23:19:05.330673 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:05.330274    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:19:05.346395 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:05.346346    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-8smpb" podStartSLOduration=2.898007183 podStartE2EDuration="5.346326371s" totalImagesPullingTime="2.448319188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:05.345084952 +0000 UTC m=+478.389461113" watchObservedRunningTime="2026-09-17 23:19:05.346326371 +0000 UTC m=+478.390702531"
Sep 17 23:19:16.335931 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:19:16.335904    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-8smpb"
Sep 17 23:22:02.597930 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.597892    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"]
Sep 17 23:22:02.601368 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.601352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.603840 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.603817    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 17 23:22:02.603840 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.603833    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:22:02.603993 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.603884    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:22:02.604131 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.604117    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:22:02.604173 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.604140    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:22:02.604213 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.604118    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 23:22:02.604660 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.604643    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:22:02.610855 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.610835    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"]
Sep 17 23:22:02.763660 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.763606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b997a1-58d8-42cb-8763-2e66fc58ff44-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.763858 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.763671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20b997a1-58d8-42cb-8763-2e66fc58ff44-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.763858 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.763717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvhxf\" (UniqueName: \"kubernetes.io/projected/20b997a1-58d8-42cb-8763-2e66fc58ff44-kube-api-access-bvhxf\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.763858 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.763755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20b997a1-58d8-42cb-8763-2e66fc58ff44-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.763858 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.763786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20b997a1-58d8-42cb-8763-2e66fc58ff44-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.864877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.864788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvhxf\" (UniqueName: \"kubernetes.io/projected/20b997a1-58d8-42cb-8763-2e66fc58ff44-kube-api-access-bvhxf\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.864877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.864837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20b997a1-58d8-42cb-8763-2e66fc58ff44-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.865092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.865078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20b997a1-58d8-42cb-8763-2e66fc58ff44-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.865145 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.865132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b997a1-58d8-42cb-8763-2e66fc58ff44-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.865187 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.865170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20b997a1-58d8-42cb-8763-2e66fc58ff44-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.865504 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.865484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b997a1-58d8-42cb-8763-2e66fc58ff44-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.865555 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.865485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20b997a1-58d8-42cb-8763-2e66fc58ff44-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.865792 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.865763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20b997a1-58d8-42cb-8763-2e66fc58ff44-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.867803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.867784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20b997a1-58d8-42cb-8763-2e66fc58ff44-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.873962 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.873946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvhxf\" (UniqueName: \"kubernetes.io/projected/20b997a1-58d8-42cb-8763-2e66fc58ff44-kube-api-access-bvhxf\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t2n8l\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:02.911911 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:02.911889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:03.189070 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:03.188992    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"]
Sep 17 23:22:03.194047 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:22:03.194017    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b997a1_58d8_42cb_8763_2e66fc58ff44.slice/crio-e600e82e7b94d23e783857da466edb5a049ce9fe86c69f8047bdd050416f9d83 WatchSource:0}: Error finding container e600e82e7b94d23e783857da466edb5a049ce9fe86c69f8047bdd050416f9d83: Status 404 returned error can't find the container with id e600e82e7b94d23e783857da466edb5a049ce9fe86c69f8047bdd050416f9d83
Sep 17 23:22:03.196593 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:03.196575    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:22:03.928355 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:03.928319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" event={"ID":"20b997a1-58d8-42cb-8763-2e66fc58ff44","Type":"ContainerStarted","Data":"e600e82e7b94d23e783857da466edb5a049ce9fe86c69f8047bdd050416f9d83"}
Sep 17 23:22:06.941908 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:06.941824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" event={"ID":"20b997a1-58d8-42cb-8763-2e66fc58ff44","Type":"ContainerStarted","Data":"4f0287105b5ab9f8d1266ac5b545b2b30a478553c1e88a0470f91a383c77d409"}
Sep 17 23:22:10.961089 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:10.961049    2580 generic.go:356] "Generic (PLEG): container finished" podID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerID="4f0287105b5ab9f8d1266ac5b545b2b30a478553c1e88a0470f91a383c77d409" exitCode=0
Sep 17 23:22:10.961449 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:10.961122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" event={"ID":"20b997a1-58d8-42cb-8763-2e66fc58ff44","Type":"ContainerDied","Data":"4f0287105b5ab9f8d1266ac5b545b2b30a478553c1e88a0470f91a383c77d409"}
Sep 17 23:22:32.054811 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:32.054780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" event={"ID":"20b997a1-58d8-42cb-8763-2e66fc58ff44","Type":"ContainerStarted","Data":"eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b"}
Sep 17 23:22:35.068117 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:35.068083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" event={"ID":"20b997a1-58d8-42cb-8763-2e66fc58ff44","Type":"ContainerStarted","Data":"3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1"}
Sep 17 23:22:35.068531 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:35.068285    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:35.068531 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:35.068433    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:35.069551 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:35.069517    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 23:22:35.088924 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:35.088880    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podStartSLOduration=2.32294198 podStartE2EDuration="33.088865686s" totalImagesPullingTime="26.658308107s" totalInitContainerRuntime="4.107615599s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:35.088026074 +0000 UTC m=+688.132402234" watchObservedRunningTime="2026-09-17 23:22:35.088865686 +0000 UTC m=+688.133241844"
Sep 17 23:22:36.071975 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:36.071930    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 23:22:41.076481 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:41.076455    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:22:41.077049 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:41.077023    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 23:22:51.077833 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:22:51.077794    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 23:23:01.077471 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:01.077386    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 23:23:11.077848 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:11.077808    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 23:23:21.077850 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:21.077811    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 23:23:31.077310 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:31.077265    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 23:23:41.077127 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:41.077083    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 23:23:46.497276 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:46.497203    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:23:53.082131 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:53.082099    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"]
Sep 17 23:23:53.082646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:53.082347    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container" containerID="cri-o://eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b" gracePeriod=30
Sep 17 23:23:53.082646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:53.082367    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kube-rbac-proxy" containerID="cri-o://3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1" gracePeriod=30
Sep 17 23:23:53.335796 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:53.335715    2580 generic.go:356] "Generic (PLEG): container finished" podID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerID="3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1" exitCode=2
Sep 17 23:23:53.335796 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:53.335776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" event={"ID":"20b997a1-58d8-42cb-8763-2e66fc58ff44","Type":"ContainerDied","Data":"3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1"}
Sep 17 23:23:56.072231 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:56.072189    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
Sep 17 23:23:56.496727 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:56.496605    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 23:23:58.014564 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.014540    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:23:58.141524 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.141455    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b997a1-58d8-42cb-8763-2e66fc58ff44-kserve-provision-location\") pod \"20b997a1-58d8-42cb-8763-2e66fc58ff44\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") "
Sep 17 23:23:58.141524 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.141494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20b997a1-58d8-42cb-8763-2e66fc58ff44-proxy-tls\") pod \"20b997a1-58d8-42cb-8763-2e66fc58ff44\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") "
Sep 17 23:23:58.141524 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.141519    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20b997a1-58d8-42cb-8763-2e66fc58ff44-cabundle-cert\") pod \"20b997a1-58d8-42cb-8763-2e66fc58ff44\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") "
Sep 17 23:23:58.141794 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.141565    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvhxf\" (UniqueName: \"kubernetes.io/projected/20b997a1-58d8-42cb-8763-2e66fc58ff44-kube-api-access-bvhxf\") pod \"20b997a1-58d8-42cb-8763-2e66fc58ff44\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") "
Sep 17 23:23:58.141794 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.141639    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20b997a1-58d8-42cb-8763-2e66fc58ff44-kube-rbac-proxy-sar-config\") pod \"20b997a1-58d8-42cb-8763-2e66fc58ff44\" (UID: \"20b997a1-58d8-42cb-8763-2e66fc58ff44\") "
Sep 17 23:23:58.141897 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.141791    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b997a1-58d8-42cb-8763-2e66fc58ff44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20b997a1-58d8-42cb-8763-2e66fc58ff44" (UID: "20b997a1-58d8-42cb-8763-2e66fc58ff44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:23:58.141992 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.141966    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b997a1-58d8-42cb-8763-2e66fc58ff44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "20b997a1-58d8-42cb-8763-2e66fc58ff44" (UID: "20b997a1-58d8-42cb-8763-2e66fc58ff44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:58.142039 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.142006    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b997a1-58d8-42cb-8763-2e66fc58ff44-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "20b997a1-58d8-42cb-8763-2e66fc58ff44" (UID: "20b997a1-58d8-42cb-8763-2e66fc58ff44"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:58.143767 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.143746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b997a1-58d8-42cb-8763-2e66fc58ff44-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "20b997a1-58d8-42cb-8763-2e66fc58ff44" (UID: "20b997a1-58d8-42cb-8763-2e66fc58ff44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:23:58.143837 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.143797    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b997a1-58d8-42cb-8763-2e66fc58ff44-kube-api-access-bvhxf" (OuterVolumeSpecName: "kube-api-access-bvhxf") pod "20b997a1-58d8-42cb-8763-2e66fc58ff44" (UID: "20b997a1-58d8-42cb-8763-2e66fc58ff44"). InnerVolumeSpecName "kube-api-access-bvhxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:58.242909 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.242877    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20b997a1-58d8-42cb-8763-2e66fc58ff44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:23:58.242909 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.242909    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b997a1-58d8-42cb-8763-2e66fc58ff44-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:23:58.243085 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.242922    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20b997a1-58d8-42cb-8763-2e66fc58ff44-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:23:58.243085 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.242934    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20b997a1-58d8-42cb-8763-2e66fc58ff44-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:23:58.243085 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.242946    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvhxf\" (UniqueName: \"kubernetes.io/projected/20b997a1-58d8-42cb-8763-2e66fc58ff44-kube-api-access-bvhxf\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:23:58.264854 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.264824    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"]
Sep 17 23:23:58.265141 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.265124    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="storage-initializer"
Sep 17 23:23:58.265223 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.265144    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="storage-initializer"
Sep 17 23:23:58.265223 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.265155    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container"
Sep 17 23:23:58.265223 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.265162    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container"
Sep 17 23:23:58.265223 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.265180    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kube-rbac-proxy"
Sep 17 23:23:58.265223 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.265189    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kube-rbac-proxy"
Sep 17 23:23:58.265454 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.265247    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kserve-container"
Sep 17 23:23:58.265454 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.265256    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerName="kube-rbac-proxy"
Sep 17 23:23:58.268432 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.268412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.271017 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.270997    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 17 23:23:58.271108 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.271004    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 23:23:58.280572 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.280553    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"]
Sep 17 23:23:58.344290 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.344254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250cbb11-af23-41f4-961c-69e28353efb6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.344456 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.344303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzq8j\" (UniqueName: \"kubernetes.io/projected/250cbb11-af23-41f4-961c-69e28353efb6-kube-api-access-fzq8j\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.344456 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.344322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/250cbb11-af23-41f4-961c-69e28353efb6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.344456 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.344401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/250cbb11-af23-41f4-961c-69e28353efb6-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.344456 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.344441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/250cbb11-af23-41f4-961c-69e28353efb6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.354461 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.354435    2580 generic.go:356] "Generic (PLEG): container finished" podID="20b997a1-58d8-42cb-8763-2e66fc58ff44" containerID="eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b" exitCode=0
Sep 17 23:23:58.354589 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.354487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" event={"ID":"20b997a1-58d8-42cb-8763-2e66fc58ff44","Type":"ContainerDied","Data":"eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b"}
Sep 17 23:23:58.354589 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.354532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l" event={"ID":"20b997a1-58d8-42cb-8763-2e66fc58ff44","Type":"ContainerDied","Data":"e600e82e7b94d23e783857da466edb5a049ce9fe86c69f8047bdd050416f9d83"}
Sep 17 23:23:58.354589 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.354489    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"
Sep 17 23:23:58.354589 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.354566    2580 scope.go:117] "RemoveContainer" containerID="3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1"
Sep 17 23:23:58.362985 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.362967    2580 scope.go:117] "RemoveContainer" containerID="eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b"
Sep 17 23:23:58.370032 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.370014    2580 scope.go:117] "RemoveContainer" containerID="4f0287105b5ab9f8d1266ac5b545b2b30a478553c1e88a0470f91a383c77d409"
Sep 17 23:23:58.377421 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.377402    2580 scope.go:117] "RemoveContainer" containerID="3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1"
Sep 17 23:23:58.377593 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.377576    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"]
Sep 17 23:23:58.377740 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:23:58.377719    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1\": container with ID starting with 3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1 not found: ID does not exist" containerID="3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1"
Sep 17 23:23:58.377829 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.377750    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1"} err="failed to get container status \"3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1\": rpc error: code = NotFound desc = could not find container \"3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1\": container with ID starting with 3914c6c2a458f91a5043f734cc02e67a6c81094e5b25ec4c59b4d7cd9bc301e1 not found: ID does not exist"
Sep 17 23:23:58.377829 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.377783    2580 scope.go:117] "RemoveContainer" containerID="eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b"
Sep 17 23:23:58.378048 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:23:58.378032    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b\": container with ID starting with eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b not found: ID does not exist" containerID="eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b"
Sep 17 23:23:58.378096 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.378054    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b"} err="failed to get container status \"eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b\": rpc error: code = NotFound desc = could not find container \"eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b\": container with ID starting with eda03a8c64cd6f20458916b1235f37b64bf549fb876507a94601107a1918526b not found: ID does not exist"
Sep 17 23:23:58.378096 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.378071    2580 scope.go:117] "RemoveContainer" containerID="4f0287105b5ab9f8d1266ac5b545b2b30a478553c1e88a0470f91a383c77d409"
Sep 17 23:23:58.378318 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:23:58.378300    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0287105b5ab9f8d1266ac5b545b2b30a478553c1e88a0470f91a383c77d409\": container with ID starting with 4f0287105b5ab9f8d1266ac5b545b2b30a478553c1e88a0470f91a383c77d409 not found: ID does not exist" containerID="4f0287105b5ab9f8d1266ac5b545b2b30a478553c1e88a0470f91a383c77d409"
Sep 17 23:23:58.378372 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.378326    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0287105b5ab9f8d1266ac5b545b2b30a478553c1e88a0470f91a383c77d409"} err="failed to get container status \"4f0287105b5ab9f8d1266ac5b545b2b30a478553c1e88a0470f91a383c77d409\": rpc error: code = NotFound desc = could not find container \"4f0287105b5ab9f8d1266ac5b545b2b30a478553c1e88a0470f91a383c77d409\": container with ID starting with 4f0287105b5ab9f8d1266ac5b545b2b30a478553c1e88a0470f91a383c77d409 not found: ID does not exist"
Sep 17 23:23:58.383467 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.383448    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t2n8l"]
Sep 17 23:23:58.445165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.445098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/250cbb11-af23-41f4-961c-69e28353efb6-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.445165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.445134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/250cbb11-af23-41f4-961c-69e28353efb6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.445341 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.445170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250cbb11-af23-41f4-961c-69e28353efb6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.445341 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.445205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzq8j\" (UniqueName: \"kubernetes.io/projected/250cbb11-af23-41f4-961c-69e28353efb6-kube-api-access-fzq8j\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.445341 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.445220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/250cbb11-af23-41f4-961c-69e28353efb6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.445487 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:23:58.445341    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 17 23:23:58.445487 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:23:58.445404    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/250cbb11-af23-41f4-961c-69e28353efb6-proxy-tls podName:250cbb11-af23-41f4-961c-69e28353efb6 nodeName:}" failed. No retries permitted until 2026-09-17 23:23:58.945387185 +0000 UTC m=+771.989763328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/250cbb11-af23-41f4-961c-69e28353efb6-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-pn6tf" (UID: "250cbb11-af23-41f4-961c-69e28353efb6") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 17 23:23:58.445597 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.445571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/250cbb11-af23-41f4-961c-69e28353efb6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.445801 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.445781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/250cbb11-af23-41f4-961c-69e28353efb6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.445958 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.445940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/250cbb11-af23-41f4-961c-69e28353efb6-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.454042 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.454020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzq8j\" (UniqueName: \"kubernetes.io/projected/250cbb11-af23-41f4-961c-69e28353efb6-kube-api-access-fzq8j\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.949756 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:58.949713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250cbb11-af23-41f4-961c-69e28353efb6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:58.949914 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:23:58.949824    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 17 23:23:58.949914 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:23:58.949871    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/250cbb11-af23-41f4-961c-69e28353efb6-proxy-tls podName:250cbb11-af23-41f4-961c-69e28353efb6 nodeName:}" failed. No retries permitted until 2026-09-17 23:23:59.949857816 +0000 UTC m=+772.994233953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/250cbb11-af23-41f4-961c-69e28353efb6-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-pn6tf" (UID: "250cbb11-af23-41f4-961c-69e28353efb6") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 17 23:23:59.500730 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:59.500689    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20b997a1-58d8-42cb-8763-2e66fc58ff44" path="/var/lib/kubelet/pods/20b997a1-58d8-42cb-8763-2e66fc58ff44/volumes"
Sep 17 23:23:59.956790 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:59.956759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250cbb11-af23-41f4-961c-69e28353efb6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:23:59.959205 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:23:59.959181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250cbb11-af23-41f4-961c-69e28353efb6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-pn6tf\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:24:00.077504 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:00.077472    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:24:00.348467 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:00.348439    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"]
Sep 17 23:24:00.351882 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:24:00.351854    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250cbb11_af23_41f4_961c_69e28353efb6.slice/crio-caf6422438f0af0fcea7b94316c12ad626e07b7f00644497489b2f867adf0aad WatchSource:0}: Error finding container caf6422438f0af0fcea7b94316c12ad626e07b7f00644497489b2f867adf0aad: Status 404 returned error can't find the container with id caf6422438f0af0fcea7b94316c12ad626e07b7f00644497489b2f867adf0aad
Sep 17 23:24:00.363903 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:00.363878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" event={"ID":"250cbb11-af23-41f4-961c-69e28353efb6","Type":"ContainerStarted","Data":"caf6422438f0af0fcea7b94316c12ad626e07b7f00644497489b2f867adf0aad"}
Sep 17 23:24:01.370228 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:01.370188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" event={"ID":"250cbb11-af23-41f4-961c-69e28353efb6","Type":"ContainerStarted","Data":"1ae09dcab4c8272b3594302eeaf74bc78b353340785c78019ee7db978ea103a7"}
Sep 17 23:24:05.385551 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:05.385520    2580 generic.go:356] "Generic (PLEG): container finished" podID="250cbb11-af23-41f4-961c-69e28353efb6" containerID="1ae09dcab4c8272b3594302eeaf74bc78b353340785c78019ee7db978ea103a7" exitCode=0
Sep 17 23:24:05.385940 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:05.385595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" event={"ID":"250cbb11-af23-41f4-961c-69e28353efb6","Type":"ContainerDied","Data":"1ae09dcab4c8272b3594302eeaf74bc78b353340785c78019ee7db978ea103a7"}
Sep 17 23:24:06.391596 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:06.391555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" event={"ID":"250cbb11-af23-41f4-961c-69e28353efb6","Type":"ContainerStarted","Data":"7d2714078df0cdd024bf2a36287ba07ad79deb17816f4f4bd70bd575321784f9"}
Sep 17 23:24:06.392086 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:06.391682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" event={"ID":"250cbb11-af23-41f4-961c-69e28353efb6","Type":"ContainerStarted","Data":"f9cdee5195ab5e06222b103aa7eadadc319ac77e28b4e942b18be0b271bdd2b7"}
Sep 17 23:24:06.392086 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:06.391905    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:24:06.411534 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:06.411487    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podStartSLOduration=4.150254836 podStartE2EDuration="8.411472101s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.261217265s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:06.410564651 +0000 UTC m=+779.454940810" watchObservedRunningTime="2026-09-17 23:24:06.411472101 +0000 UTC m=+779.455848260"
Sep 17 23:24:07.394823 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:07.394793    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:24:07.396110 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:07.396086    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 23:24:08.398333 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:08.398297    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 23:24:13.402967 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:13.402938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:24:13.403634 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:13.403585    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 23:24:23.404279 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:23.404241    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 23:24:33.403919 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:33.403834    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 23:24:43.404366 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:43.404328    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 23:24:53.404289 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:24:53.404252    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 23:25:03.403917 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:03.403876    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 23:25:13.404444 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:13.404403    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 23:25:23.404594 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:23.404566    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:25:28.781499 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:28.781468    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"]
Sep 17 23:25:28.782098 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:28.781747    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container" containerID="cri-o://f9cdee5195ab5e06222b103aa7eadadc319ac77e28b4e942b18be0b271bdd2b7" gracePeriod=30
Sep 17 23:25:28.782098 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:28.781758    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kube-rbac-proxy" containerID="cri-o://7d2714078df0cdd024bf2a36287ba07ad79deb17816f4f4bd70bd575321784f9" gracePeriod=30
Sep 17 23:25:29.691743 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:29.691708    2580 generic.go:356] "Generic (PLEG): container finished" podID="250cbb11-af23-41f4-961c-69e28353efb6" containerID="7d2714078df0cdd024bf2a36287ba07ad79deb17816f4f4bd70bd575321784f9" exitCode=2
Sep 17 23:25:29.691926 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:29.691783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" event={"ID":"250cbb11-af23-41f4-961c-69e28353efb6","Type":"ContainerDied","Data":"7d2714078df0cdd024bf2a36287ba07ad79deb17816f4f4bd70bd575321784f9"}
Sep 17 23:25:33.398966 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.398910    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 17 23:25:33.404246 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.404217    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 23:25:33.707487 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.707452    2580 generic.go:356] "Generic (PLEG): container finished" podID="250cbb11-af23-41f4-961c-69e28353efb6" containerID="f9cdee5195ab5e06222b103aa7eadadc319ac77e28b4e942b18be0b271bdd2b7" exitCode=0
Sep 17 23:25:33.707623 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.707518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" event={"ID":"250cbb11-af23-41f4-961c-69e28353efb6","Type":"ContainerDied","Data":"f9cdee5195ab5e06222b103aa7eadadc319ac77e28b4e942b18be0b271bdd2b7"}
Sep 17 23:25:33.814645 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.814608    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:25:33.924563 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.924484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzq8j\" (UniqueName: \"kubernetes.io/projected/250cbb11-af23-41f4-961c-69e28353efb6-kube-api-access-fzq8j\") pod \"250cbb11-af23-41f4-961c-69e28353efb6\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") "
Sep 17 23:25:33.924563 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.924522    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250cbb11-af23-41f4-961c-69e28353efb6-proxy-tls\") pod \"250cbb11-af23-41f4-961c-69e28353efb6\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") "
Sep 17 23:25:33.924563 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.924541    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/250cbb11-af23-41f4-961c-69e28353efb6-kserve-provision-location\") pod \"250cbb11-af23-41f4-961c-69e28353efb6\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") "
Sep 17 23:25:33.924856 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.924572    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/250cbb11-af23-41f4-961c-69e28353efb6-kube-rbac-proxy-sar-config\") pod \"250cbb11-af23-41f4-961c-69e28353efb6\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") "
Sep 17 23:25:33.924856 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.924598    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/250cbb11-af23-41f4-961c-69e28353efb6-cabundle-cert\") pod \"250cbb11-af23-41f4-961c-69e28353efb6\" (UID: \"250cbb11-af23-41f4-961c-69e28353efb6\") "
Sep 17 23:25:33.924957 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.924863    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250cbb11-af23-41f4-961c-69e28353efb6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "250cbb11-af23-41f4-961c-69e28353efb6" (UID: "250cbb11-af23-41f4-961c-69e28353efb6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:33.924957 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.924937    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250cbb11-af23-41f4-961c-69e28353efb6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "250cbb11-af23-41f4-961c-69e28353efb6" (UID: "250cbb11-af23-41f4-961c-69e28353efb6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:33.925061 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.925039    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250cbb11-af23-41f4-961c-69e28353efb6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "250cbb11-af23-41f4-961c-69e28353efb6" (UID: "250cbb11-af23-41f4-961c-69e28353efb6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:33.926799 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.926776    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250cbb11-af23-41f4-961c-69e28353efb6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "250cbb11-af23-41f4-961c-69e28353efb6" (UID: "250cbb11-af23-41f4-961c-69e28353efb6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:25:33.926888 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.926814    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250cbb11-af23-41f4-961c-69e28353efb6-kube-api-access-fzq8j" (OuterVolumeSpecName: "kube-api-access-fzq8j") pod "250cbb11-af23-41f4-961c-69e28353efb6" (UID: "250cbb11-af23-41f4-961c-69e28353efb6"). InnerVolumeSpecName "kube-api-access-fzq8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:33.957454 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.957426    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"]
Sep 17 23:25:33.957753 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.957740    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kube-rbac-proxy"
Sep 17 23:25:33.957797 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.957756    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kube-rbac-proxy"
Sep 17 23:25:33.957797 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.957766    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="storage-initializer"
Sep 17 23:25:33.957797 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.957771    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="storage-initializer"
Sep 17 23:25:33.957797 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.957782    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container"
Sep 17 23:25:33.957797 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.957787    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container"
Sep 17 23:25:33.957940 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.957834    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kube-rbac-proxy"
Sep 17 23:25:33.957940 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.957842    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="250cbb11-af23-41f4-961c-69e28353efb6" containerName="kserve-container"
Sep 17 23:25:33.960919 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.960905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:33.963131 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.963110    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 17 23:25:33.963241 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.963149    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 23:25:33.969002 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:33.968983    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"]
Sep 17 23:25:34.026018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.025973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.026208 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.026045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.026208 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.026076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.026208 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.026102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.026208 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.026160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsfrl\" (UniqueName: \"kubernetes.io/projected/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kube-api-access-dsfrl\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.026208 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.026193    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/250cbb11-af23-41f4-961c-69e28353efb6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:25:34.026208 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.026203    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/250cbb11-af23-41f4-961c-69e28353efb6-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:25:34.026208 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.026213    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzq8j\" (UniqueName: \"kubernetes.io/projected/250cbb11-af23-41f4-961c-69e28353efb6-kube-api-access-fzq8j\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:25:34.026425 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.026221    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250cbb11-af23-41f4-961c-69e28353efb6-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:25:34.026425 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.026229    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/250cbb11-af23-41f4-961c-69e28353efb6-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:25:34.127216 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.127183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.127377 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.127230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.127377 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.127289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsfrl\" (UniqueName: \"kubernetes.io/projected/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kube-api-access-dsfrl\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.127377 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.127326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.127377 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.127362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.127558 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:25:34.127424    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 17 23:25:34.127558 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:25:34.127485    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-proxy-tls podName:dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f nodeName:}" failed. No retries permitted until 2026-09-17 23:25:34.627466953 +0000 UTC m=+867.671843092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" (UID: "dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 17 23:25:34.127862 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.127840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.127931 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.127876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.128158 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.128141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.137826 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.137798    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsfrl\" (UniqueName: \"kubernetes.io/projected/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kube-api-access-dsfrl\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.631464 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.631421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.633914 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.633894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:34.712921 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.712891    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"
Sep 17 23:25:34.713103 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.712885    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf" event={"ID":"250cbb11-af23-41f4-961c-69e28353efb6","Type":"ContainerDied","Data":"caf6422438f0af0fcea7b94316c12ad626e07b7f00644497489b2f867adf0aad"}
Sep 17 23:25:34.713103 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.713043    2580 scope.go:117] "RemoveContainer" containerID="7d2714078df0cdd024bf2a36287ba07ad79deb17816f4f4bd70bd575321784f9"
Sep 17 23:25:34.720880 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.720862    2580 scope.go:117] "RemoveContainer" containerID="f9cdee5195ab5e06222b103aa7eadadc319ac77e28b4e942b18be0b271bdd2b7"
Sep 17 23:25:34.727793 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.727779    2580 scope.go:117] "RemoveContainer" containerID="1ae09dcab4c8272b3594302eeaf74bc78b353340785c78019ee7db978ea103a7"
Sep 17 23:25:34.733262 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.733241    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"]
Sep 17 23:25:34.735822 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.735806    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pn6tf"]
Sep 17 23:25:34.870479 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:34.870445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:35.141040 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:35.141010    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"]
Sep 17 23:25:35.144172 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:25:35.144146    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9cab6b_2f3c_4a5d_bd8c_9b631463b05f.slice/crio-695449a9d23b4eb9efd7135d99a527f7d0c269618d3d6f2095960fd11571003f WatchSource:0}: Error finding container 695449a9d23b4eb9efd7135d99a527f7d0c269618d3d6f2095960fd11571003f: Status 404 returned error can't find the container with id 695449a9d23b4eb9efd7135d99a527f7d0c269618d3d6f2095960fd11571003f
Sep 17 23:25:35.503894 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:35.503817    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="250cbb11-af23-41f4-961c-69e28353efb6" path="/var/lib/kubelet/pods/250cbb11-af23-41f4-961c-69e28353efb6/volumes"
Sep 17 23:25:35.719542 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:35.719507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" event={"ID":"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f","Type":"ContainerStarted","Data":"4ce48395182185dd6b39f1431ab43e6688c46db58d970381d11666a73ddcb4d3"}
Sep 17 23:25:35.719922 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:35.719561    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" event={"ID":"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f","Type":"ContainerStarted","Data":"695449a9d23b4eb9efd7135d99a527f7d0c269618d3d6f2095960fd11571003f"}
Sep 17 23:25:39.734764 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:39.734729    2580 generic.go:356] "Generic (PLEG): container finished" podID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerID="4ce48395182185dd6b39f1431ab43e6688c46db58d970381d11666a73ddcb4d3" exitCode=0
Sep 17 23:25:39.735322 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:39.734810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" event={"ID":"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f","Type":"ContainerDied","Data":"4ce48395182185dd6b39f1431ab43e6688c46db58d970381d11666a73ddcb4d3"}
Sep 17 23:25:40.740689 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:40.740658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" event={"ID":"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f","Type":"ContainerStarted","Data":"90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080"}
Sep 17 23:25:40.741071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:40.740714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" event={"ID":"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f","Type":"ContainerStarted","Data":"0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791"}
Sep 17 23:25:40.741071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:40.740994    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:40.741160 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:40.741144    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:40.742389 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:40.742365    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 23:25:40.758211 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:40.758169    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podStartSLOduration=3.842061695 podStartE2EDuration="7.758154745s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.91609305s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:40.757549422 +0000 UTC m=+873.801925583" watchObservedRunningTime="2026-09-17 23:25:40.758154745 +0000 UTC m=+873.802530895"
Sep 17 23:25:41.744459 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:41.744416    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 23:25:46.748394 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:46.748356    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:25:46.748956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:46.748931    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 23:25:56.748956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:25:56.748917    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 23:26:06.749600 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:26:06.749517    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 23:26:16.748933 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:26:16.748895    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 23:26:26.749452 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:26:26.749406    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 23:26:36.749351 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:26:36.749307    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 23:26:46.749433 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:26:46.749389    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 23:26:53.497178 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:26:53.497120    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 23:27:03.067777 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.067734    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"]
Sep 17 23:27:03.071081 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.071060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.073408 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.073389    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 17 23:27:03.073508 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.073398    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 17 23:27:03.081524 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.081506    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"]
Sep 17 23:27:03.166094 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.166052    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.166094 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.166102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.166320 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.166142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88xqs\" (UniqueName: \"kubernetes.io/projected/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kube-api-access-88xqs\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.166320 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.166165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.166320 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.166182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.267166 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.267131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.267351 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.267181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.267351 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.267223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88xqs\" (UniqueName: \"kubernetes.io/projected/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kube-api-access-88xqs\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.267351 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.267251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.267351 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.267277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.267756 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.267733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.267978 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.267949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.268051 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.268015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.269781 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.269763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.275512 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.275488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88xqs\" (UniqueName: \"kubernetes.io/projected/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kube-api-access-88xqs\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rv5c2\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.381027 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.380937    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:27:03.500604 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.500581    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:27:03.651359 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:03.651282    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"]
Sep 17 23:27:03.654362 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:27:03.654326    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658b81cb_07a6_44c5_9a49_fbcbcbdcaf43.slice/crio-35fcfbb764b54825874375b4eb085de0d9a6c471f9126f19f5f9247af989161e WatchSource:0}: Error finding container 35fcfbb764b54825874375b4eb085de0d9a6c471f9126f19f5f9247af989161e: Status 404 returned error can't find the container with id 35fcfbb764b54825874375b4eb085de0d9a6c471f9126f19f5f9247af989161e
Sep 17 23:27:04.035474 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:04.035437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" event={"ID":"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43","Type":"ContainerStarted","Data":"6c67e2cf76131343de3a813d98f055c9b6964daf66fbd3cda6590bed7053a0e4"}
Sep 17 23:27:04.035691 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:04.035502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" event={"ID":"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43","Type":"ContainerStarted","Data":"35fcfbb764b54825874375b4eb085de0d9a6c471f9126f19f5f9247af989161e"}
Sep 17 23:27:16.586731 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:16.586697    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"]
Sep 17 23:27:16.587106 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:16.586998    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container" containerID="cri-o://0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791" gracePeriod=30
Sep 17 23:27:16.587106 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:16.587022    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kube-rbac-proxy" containerID="cri-o://90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080" gracePeriod=30
Sep 17 23:27:16.744898 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:16.744855    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 17 23:27:17.081825 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:17.081795    2580 generic.go:356] "Generic (PLEG): container finished" podID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerID="90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080" exitCode=2
Sep 17 23:27:17.082001 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:17.081867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" event={"ID":"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f","Type":"ContainerDied","Data":"90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080"}
Sep 17 23:27:21.745218 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:21.745179    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 17 23:27:22.329566 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.329542    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:27:22.410187 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.410155    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kserve-provision-location\") pod \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") "
Sep 17 23:27:22.410344 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.410204    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-cabundle-cert\") pod \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") "
Sep 17 23:27:22.410344 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.410238    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsfrl\" (UniqueName: \"kubernetes.io/projected/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kube-api-access-dsfrl\") pod \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") "
Sep 17 23:27:22.410344 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.410261    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kube-rbac-proxy-sar-config\") pod \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") "
Sep 17 23:27:22.410344 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.410305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-proxy-tls\") pod \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\" (UID: \"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f\") "
Sep 17 23:27:22.410564 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.410534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" (UID: "dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:22.410651 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.410595    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" (UID: "dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:22.410651 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.410604    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" (UID: "dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:22.412353 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.412330    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kube-api-access-dsfrl" (OuterVolumeSpecName: "kube-api-access-dsfrl") pod "dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" (UID: "dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f"). InnerVolumeSpecName "kube-api-access-dsfrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:22.412428 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.412385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" (UID: "dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:22.511185 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.511097    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:27:22.511185 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.511125    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:27:22.511185 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.511138    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsfrl\" (UniqueName: \"kubernetes.io/projected/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kube-api-access-dsfrl\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:27:22.511185 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.511148    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:27:22.511185 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:22.511160    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:27:23.102748 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.102711    2580 generic.go:356] "Generic (PLEG): container finished" podID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerID="0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791" exitCode=0
Sep 17 23:27:23.103119 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.102767    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"
Sep 17 23:27:23.103119 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.102798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" event={"ID":"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f","Type":"ContainerDied","Data":"0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791"}
Sep 17 23:27:23.103119 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.102860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh" event={"ID":"dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f","Type":"ContainerDied","Data":"695449a9d23b4eb9efd7135d99a527f7d0c269618d3d6f2095960fd11571003f"}
Sep 17 23:27:23.103119 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.102889    2580 scope.go:117] "RemoveContainer" containerID="90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080"
Sep 17 23:27:23.111174 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.111155    2580 scope.go:117] "RemoveContainer" containerID="0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791"
Sep 17 23:27:23.118156 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.118139    2580 scope.go:117] "RemoveContainer" containerID="4ce48395182185dd6b39f1431ab43e6688c46db58d970381d11666a73ddcb4d3"
Sep 17 23:27:23.123880 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.123858    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"]
Sep 17 23:27:23.125147 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.125134    2580 scope.go:117] "RemoveContainer" containerID="90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080"
Sep 17 23:27:23.125385 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:27:23.125368    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080\": container with ID starting with 90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080 not found: ID does not exist" containerID="90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080"
Sep 17 23:27:23.125444 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.125395    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080"} err="failed to get container status \"90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080\": rpc error: code = NotFound desc = could not find container \"90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080\": container with ID starting with 90e7191b50ce3c39190f409e4dea747f07c4e4d79a875ce1f4d9ad2f59bd9080 not found: ID does not exist"
Sep 17 23:27:23.125444 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.125410    2580 scope.go:117] "RemoveContainer" containerID="0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791"
Sep 17 23:27:23.125641 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:27:23.125625    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791\": container with ID starting with 0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791 not found: ID does not exist" containerID="0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791"
Sep 17 23:27:23.125685 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.125645    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791"} err="failed to get container status \"0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791\": rpc error: code = NotFound desc = could not find container \"0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791\": container with ID starting with 0df2e809d2f7c9ec63170b47c3ece403c38301c9e45a3edbff4556405dd9a791 not found: ID does not exist"
Sep 17 23:27:23.125685 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.125657    2580 scope.go:117] "RemoveContainer" containerID="4ce48395182185dd6b39f1431ab43e6688c46db58d970381d11666a73ddcb4d3"
Sep 17 23:27:23.125879 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:27:23.125863    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce48395182185dd6b39f1431ab43e6688c46db58d970381d11666a73ddcb4d3\": container with ID starting with 4ce48395182185dd6b39f1431ab43e6688c46db58d970381d11666a73ddcb4d3 not found: ID does not exist" containerID="4ce48395182185dd6b39f1431ab43e6688c46db58d970381d11666a73ddcb4d3"
Sep 17 23:27:23.125926 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.125882    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce48395182185dd6b39f1431ab43e6688c46db58d970381d11666a73ddcb4d3"} err="failed to get container status \"4ce48395182185dd6b39f1431ab43e6688c46db58d970381d11666a73ddcb4d3\": rpc error: code = NotFound desc = could not find container \"4ce48395182185dd6b39f1431ab43e6688c46db58d970381d11666a73ddcb4d3\": container with ID starting with 4ce48395182185dd6b39f1431ab43e6688c46db58d970381d11666a73ddcb4d3 not found: ID does not exist"
Sep 17 23:27:23.129769 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.129750    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rqpqh"]
Sep 17 23:27:23.500897 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:23.500820    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" path="/var/lib/kubelet/pods/dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f/volumes"
Sep 17 23:27:34.142630 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:34.142587    2580 generic.go:356] "Generic (PLEG): container finished" podID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerID="6c67e2cf76131343de3a813d98f055c9b6964daf66fbd3cda6590bed7053a0e4" exitCode=0
Sep 17 23:27:34.143018 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:34.142650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" event={"ID":"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43","Type":"ContainerDied","Data":"6c67e2cf76131343de3a813d98f055c9b6964daf66fbd3cda6590bed7053a0e4"}
Sep 17 23:27:34.143942 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:27:34.143925    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:29:22.572839 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:22.572759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" event={"ID":"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43","Type":"ContainerStarted","Data":"7fe1c546dd5dd8e5d0ec0505af1c021993e53f7bbaeb998de7cae4d0e15d062c"}
Sep 17 23:29:22.573244 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:22.572872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" event={"ID":"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43","Type":"ContainerStarted","Data":"c7b5170794aff6cd036eb83f67d0411cded5c9b189cea64a1dd2d5cef29fa6bc"}
Sep 17 23:29:22.573244 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:22.573032    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:29:22.573244 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:22.573085    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:29:22.574557 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:22.574527    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:29:22.592065 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:22.591955    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" podStartSLOduration=2.095123897 podStartE2EDuration="2m19.591928495s" totalImagesPullingTime="1m47.90197165s" totalInitContainerRuntime="29.594832948s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:29:22.591413945 +0000 UTC m=+1095.635790114" watchObservedRunningTime="2026-09-17 23:29:22.591928495 +0000 UTC m=+1095.636304655"
Sep 17 23:29:23.576444 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:23.576405    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:29:28.580592 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:28.580562    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:29:28.581107 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:28.581081    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:29:38.581605 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:38.581576    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:29:49.220139 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:49.220108    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"]
Sep 17 23:29:49.220644 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:49.220338    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="kserve-container" containerID="cri-o://c7b5170794aff6cd036eb83f67d0411cded5c9b189cea64a1dd2d5cef29fa6bc" gracePeriod=30
Sep 17 23:29:49.220644 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:49.220382    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="kube-rbac-proxy" containerID="cri-o://7fe1c546dd5dd8e5d0ec0505af1c021993e53f7bbaeb998de7cae4d0e15d062c" gracePeriod=30
Sep 17 23:29:49.665412 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:49.665381    2580 generic.go:356] "Generic (PLEG): container finished" podID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerID="7fe1c546dd5dd8e5d0ec0505af1c021993e53f7bbaeb998de7cae4d0e15d062c" exitCode=2
Sep 17 23:29:49.665412 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:49.665423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" event={"ID":"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43","Type":"ContainerDied","Data":"7fe1c546dd5dd8e5d0ec0505af1c021993e53f7bbaeb998de7cae4d0e15d062c"}
Sep 17 23:29:50.670608 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.670579    2580 generic.go:356] "Generic (PLEG): container finished" podID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerID="c7b5170794aff6cd036eb83f67d0411cded5c9b189cea64a1dd2d5cef29fa6bc" exitCode=0
Sep 17 23:29:50.670964 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.670650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" event={"ID":"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43","Type":"ContainerDied","Data":"c7b5170794aff6cd036eb83f67d0411cded5c9b189cea64a1dd2d5cef29fa6bc"}
Sep 17 23:29:50.670964 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.670727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2" event={"ID":"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43","Type":"ContainerDied","Data":"35fcfbb764b54825874375b4eb085de0d9a6c471f9126f19f5f9247af989161e"}
Sep 17 23:29:50.670964 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.670762    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35fcfbb764b54825874375b4eb085de0d9a6c471f9126f19f5f9247af989161e"
Sep 17 23:29:50.680104 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.680084    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:29:50.787896 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.787868    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-cabundle-cert\") pod \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") "
Sep 17 23:29:50.788051 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.787919    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kserve-provision-location\") pod \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") "
Sep 17 23:29:50.788051 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.787947    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88xqs\" (UniqueName: \"kubernetes.io/projected/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kube-api-access-88xqs\") pod \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") "
Sep 17 23:29:50.788051 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.787976    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-proxy-tls\") pod \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") "
Sep 17 23:29:50.788051 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.788025    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kube-rbac-proxy-sar-config\") pod \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\" (UID: \"658b81cb-07a6-44c5-9a49-fbcbcbdcaf43\") "
Sep 17 23:29:50.788385 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.788271    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" (UID: "658b81cb-07a6-44c5-9a49-fbcbcbdcaf43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:29:50.788448 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.788427    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" (UID: "658b81cb-07a6-44c5-9a49-fbcbcbdcaf43"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:29:50.790142 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.790112    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" (UID: "658b81cb-07a6-44c5-9a49-fbcbcbdcaf43"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:29:50.790262 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.790240    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kube-api-access-88xqs" (OuterVolumeSpecName: "kube-api-access-88xqs") pod "658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" (UID: "658b81cb-07a6-44c5-9a49-fbcbcbdcaf43"). InnerVolumeSpecName "kube-api-access-88xqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:29:50.792356 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.792337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" (UID: "658b81cb-07a6-44c5-9a49-fbcbcbdcaf43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:50.888957 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.888926    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:29:50.888957 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.888952    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:29:50.888957 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.888962    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88xqs\" (UniqueName: \"kubernetes.io/projected/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kube-api-access-88xqs\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:29:50.889167 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.888974    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:29:50.889167 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:50.888985    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:29:51.673877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:51.673845    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"
Sep 17 23:29:51.692682 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:51.692656    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"]
Sep 17 23:29:51.696398 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:51.696368    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rv5c2"]
Sep 17 23:29:53.500597 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:53.500567    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" path="/var/lib/kubelet/pods/658b81cb-07a6-44c5-9a49-fbcbcbdcaf43/volumes"
Sep 17 23:29:54.423101 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423068    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"]
Sep 17 23:29:54.423318 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423301    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="storage-initializer"
Sep 17 23:29:54.423318 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423314    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="storage-initializer"
Sep 17 23:29:54.423393 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423321    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kube-rbac-proxy"
Sep 17 23:29:54.423393 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423326    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kube-rbac-proxy"
Sep 17 23:29:54.423393 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423336    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="kserve-container"
Sep 17 23:29:54.423393 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423342    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="kserve-container"
Sep 17 23:29:54.423393 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423347    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="kube-rbac-proxy"
Sep 17 23:29:54.423393 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423351    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="kube-rbac-proxy"
Sep 17 23:29:54.423393 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423365    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container"
Sep 17 23:29:54.423393 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423369    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container"
Sep 17 23:29:54.423393 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423374    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="storage-initializer"
Sep 17 23:29:54.423393 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423379    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="storage-initializer"
Sep 17 23:29:54.423692 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423418    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="kserve-container"
Sep 17 23:29:54.423692 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423426    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kube-rbac-proxy"
Sep 17 23:29:54.423692 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423431    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="658b81cb-07a6-44c5-9a49-fbcbcbdcaf43" containerName="kube-rbac-proxy"
Sep 17 23:29:54.423692 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.423437    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc9cab6b-2f3c-4a5d-bd8c-9b631463b05f" containerName="kserve-container"
Sep 17 23:29:54.446649 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.446598    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"]
Sep 17 23:29:54.446776 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.446719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.449061 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.449039    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:29:54.449170 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.449073    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:29:54.450129 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.450108    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 17 23:29:54.450236 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.450146    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:29:54.450236 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.450168    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 17 23:29:54.450236 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.450179    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:29:54.450236 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.450202    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:29:54.613912 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.613877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde82b04-f192-431c-baab-1352b1d7fcc9-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.613912 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.613908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v4bf\" (UniqueName: \"kubernetes.io/projected/cde82b04-f192-431c-baab-1352b1d7fcc9-kube-api-access-2v4bf\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.614314 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.613931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde82b04-f192-431c-baab-1352b1d7fcc9-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.614314 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.613991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde82b04-f192-431c-baab-1352b1d7fcc9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.614314 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.614016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde82b04-f192-431c-baab-1352b1d7fcc9-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.715402 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.715327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde82b04-f192-431c-baab-1352b1d7fcc9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.715402 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.715370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde82b04-f192-431c-baab-1352b1d7fcc9-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.715650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.715415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde82b04-f192-431c-baab-1352b1d7fcc9-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.715650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.715437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v4bf\" (UniqueName: \"kubernetes.io/projected/cde82b04-f192-431c-baab-1352b1d7fcc9-kube-api-access-2v4bf\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.715650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.715466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde82b04-f192-431c-baab-1352b1d7fcc9-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.715912 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.715891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde82b04-f192-431c-baab-1352b1d7fcc9-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.716175 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.716155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde82b04-f192-431c-baab-1352b1d7fcc9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.716218 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.716155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde82b04-f192-431c-baab-1352b1d7fcc9-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.718003 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.717985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde82b04-f192-431c-baab-1352b1d7fcc9-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.727167 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.727139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v4bf\" (UniqueName: \"kubernetes.io/projected/cde82b04-f192-431c-baab-1352b1d7fcc9-kube-api-access-2v4bf\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-tbcm9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:54.756151 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:54.756124    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:29:55.038006 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:55.037964    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"]
Sep 17 23:29:55.041295 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:29:55.041268    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde82b04_f192_431c_baab_1352b1d7fcc9.slice/crio-fa9004ea07e5d304cd123b6e94c2c846768831b24ba820194db9a2a25c878a3f WatchSource:0}: Error finding container fa9004ea07e5d304cd123b6e94c2c846768831b24ba820194db9a2a25c878a3f: Status 404 returned error can't find the container with id fa9004ea07e5d304cd123b6e94c2c846768831b24ba820194db9a2a25c878a3f
Sep 17 23:29:55.687828 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:55.687794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9" event={"ID":"cde82b04-f192-431c-baab-1352b1d7fcc9","Type":"ContainerStarted","Data":"5452e69e51c9f6296bb8c9f4dbe7fde518ceea5946f241264da8a5c5f68b42d7"}
Sep 17 23:29:55.688343 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:29:55.687853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9" event={"ID":"cde82b04-f192-431c-baab-1352b1d7fcc9","Type":"ContainerStarted","Data":"fa9004ea07e5d304cd123b6e94c2c846768831b24ba820194db9a2a25c878a3f"}
Sep 17 23:30:00.706182 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:30:00.706146    2580 generic.go:356] "Generic (PLEG): container finished" podID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerID="5452e69e51c9f6296bb8c9f4dbe7fde518ceea5946f241264da8a5c5f68b42d7" exitCode=0
Sep 17 23:30:00.706654 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:30:00.706220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9" event={"ID":"cde82b04-f192-431c-baab-1352b1d7fcc9","Type":"ContainerDied","Data":"5452e69e51c9f6296bb8c9f4dbe7fde518ceea5946f241264da8a5c5f68b42d7"}
Sep 17 23:30:01.713194 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:30:01.713161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9" event={"ID":"cde82b04-f192-431c-baab-1352b1d7fcc9","Type":"ContainerStarted","Data":"7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f"}
Sep 17 23:30:01.713708 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:30:01.713219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9" event={"ID":"cde82b04-f192-431c-baab-1352b1d7fcc9","Type":"ContainerStarted","Data":"82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e"}
Sep 17 23:30:01.713708 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:30:01.713417    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:30:01.713708 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:30:01.713540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:30:01.732719 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:30:01.732677    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9" podStartSLOduration=2.696584509 podStartE2EDuration="7.732663598s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.036079089s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:01.731462488 +0000 UTC m=+1134.775838646" watchObservedRunningTime="2026-09-17 23:30:01.732663598 +0000 UTC m=+1134.777039756"
Sep 17 23:30:07.722279 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:30:07.722247    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:31:32.529381 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:32.529352    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:31:40.557330 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:40.557296    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"]
Sep 17 23:31:40.557754 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:40.557534    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9" podUID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerName="kserve-container" containerID="cri-o://82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e" gracePeriod=30
Sep 17 23:31:40.557754 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:40.557584    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9" podUID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerName="kube-rbac-proxy" containerID="cri-o://7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f" gracePeriod=30
Sep 17 23:31:41.068985 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:41.068953    2580 generic.go:356] "Generic (PLEG): container finished" podID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerID="7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f" exitCode=2
Sep 17 23:31:41.069163 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:41.069028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9" event={"ID":"cde82b04-f192-431c-baab-1352b1d7fcc9","Type":"ContainerDied","Data":"7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f"}
Sep 17 23:31:42.004333 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.004309    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:31:42.074200 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.074116    2580 generic.go:356] "Generic (PLEG): container finished" podID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerID="82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e" exitCode=0
Sep 17 23:31:42.074200 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.074179    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"
Sep 17 23:31:42.074200 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.074192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9" event={"ID":"cde82b04-f192-431c-baab-1352b1d7fcc9","Type":"ContainerDied","Data":"82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e"}
Sep 17 23:31:42.074400 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.074267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9" event={"ID":"cde82b04-f192-431c-baab-1352b1d7fcc9","Type":"ContainerDied","Data":"fa9004ea07e5d304cd123b6e94c2c846768831b24ba820194db9a2a25c878a3f"}
Sep 17 23:31:42.074400 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.074292    2580 scope.go:117] "RemoveContainer" containerID="7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f"
Sep 17 23:31:42.082315 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.082299    2580 scope.go:117] "RemoveContainer" containerID="82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e"
Sep 17 23:31:42.089517 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.089503    2580 scope.go:117] "RemoveContainer" containerID="5452e69e51c9f6296bb8c9f4dbe7fde518ceea5946f241264da8a5c5f68b42d7"
Sep 17 23:31:42.093596 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.093576    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde82b04-f192-431c-baab-1352b1d7fcc9-cabundle-cert\") pod \"cde82b04-f192-431c-baab-1352b1d7fcc9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") "
Sep 17 23:31:42.093709 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.093638    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde82b04-f192-431c-baab-1352b1d7fcc9-kube-rbac-proxy-sar-config\") pod \"cde82b04-f192-431c-baab-1352b1d7fcc9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") "
Sep 17 23:31:42.093709 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.093700    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v4bf\" (UniqueName: \"kubernetes.io/projected/cde82b04-f192-431c-baab-1352b1d7fcc9-kube-api-access-2v4bf\") pod \"cde82b04-f192-431c-baab-1352b1d7fcc9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") "
Sep 17 23:31:42.093807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.093729    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde82b04-f192-431c-baab-1352b1d7fcc9-kserve-provision-location\") pod \"cde82b04-f192-431c-baab-1352b1d7fcc9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") "
Sep 17 23:31:42.093807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.093772    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde82b04-f192-431c-baab-1352b1d7fcc9-proxy-tls\") pod \"cde82b04-f192-431c-baab-1352b1d7fcc9\" (UID: \"cde82b04-f192-431c-baab-1352b1d7fcc9\") "
Sep 17 23:31:42.094017 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.093989    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde82b04-f192-431c-baab-1352b1d7fcc9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cde82b04-f192-431c-baab-1352b1d7fcc9" (UID: "cde82b04-f192-431c-baab-1352b1d7fcc9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:42.094066 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.094015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde82b04-f192-431c-baab-1352b1d7fcc9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cde82b04-f192-431c-baab-1352b1d7fcc9" (UID: "cde82b04-f192-431c-baab-1352b1d7fcc9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:42.095905 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.095877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde82b04-f192-431c-baab-1352b1d7fcc9-kube-api-access-2v4bf" (OuterVolumeSpecName: "kube-api-access-2v4bf") pod "cde82b04-f192-431c-baab-1352b1d7fcc9" (UID: "cde82b04-f192-431c-baab-1352b1d7fcc9"). InnerVolumeSpecName "kube-api-access-2v4bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:42.096055 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.096033    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde82b04-f192-431c-baab-1352b1d7fcc9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cde82b04-f192-431c-baab-1352b1d7fcc9" (UID: "cde82b04-f192-431c-baab-1352b1d7fcc9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:31:42.097658 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.097629    2580 scope.go:117] "RemoveContainer" containerID="7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f"
Sep 17 23:31:42.097937 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:31:42.097919    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f\": container with ID starting with 7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f not found: ID does not exist" containerID="7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f"
Sep 17 23:31:42.098016 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.097943    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f"} err="failed to get container status \"7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f\": rpc error: code = NotFound desc = could not find container \"7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f\": container with ID starting with 7b64b2ecb6cec4c11f2225f1e485370b39e8c7af1fb43f2ee96342d393ca813f not found: ID does not exist"
Sep 17 23:31:42.098016 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.097973    2580 scope.go:117] "RemoveContainer" containerID="82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e"
Sep 17 23:31:42.098249 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:31:42.098224    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e\": container with ID starting with 82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e not found: ID does not exist" containerID="82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e"
Sep 17 23:31:42.098249 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.098244    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e"} err="failed to get container status \"82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e\": rpc error: code = NotFound desc = could not find container \"82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e\": container with ID starting with 82560dcc429d6a574eb9664ad3fe7bdca671a0e9650515c76d189409ee62642e not found: ID does not exist"
Sep 17 23:31:42.098418 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.098257    2580 scope.go:117] "RemoveContainer" containerID="5452e69e51c9f6296bb8c9f4dbe7fde518ceea5946f241264da8a5c5f68b42d7"
Sep 17 23:31:42.098563 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:31:42.098540    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5452e69e51c9f6296bb8c9f4dbe7fde518ceea5946f241264da8a5c5f68b42d7\": container with ID starting with 5452e69e51c9f6296bb8c9f4dbe7fde518ceea5946f241264da8a5c5f68b42d7 not found: ID does not exist" containerID="5452e69e51c9f6296bb8c9f4dbe7fde518ceea5946f241264da8a5c5f68b42d7"
Sep 17 23:31:42.098651 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.098569    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5452e69e51c9f6296bb8c9f4dbe7fde518ceea5946f241264da8a5c5f68b42d7"} err="failed to get container status \"5452e69e51c9f6296bb8c9f4dbe7fde518ceea5946f241264da8a5c5f68b42d7\": rpc error: code = NotFound desc = could not find container \"5452e69e51c9f6296bb8c9f4dbe7fde518ceea5946f241264da8a5c5f68b42d7\": container with ID starting with 5452e69e51c9f6296bb8c9f4dbe7fde518ceea5946f241264da8a5c5f68b42d7 not found: ID does not exist"
Sep 17 23:31:42.099358 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.099338    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde82b04-f192-431c-baab-1352b1d7fcc9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cde82b04-f192-431c-baab-1352b1d7fcc9" (UID: "cde82b04-f192-431c-baab-1352b1d7fcc9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:42.194670 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.194644    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v4bf\" (UniqueName: \"kubernetes.io/projected/cde82b04-f192-431c-baab-1352b1d7fcc9-kube-api-access-2v4bf\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:31:42.194670 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.194666    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde82b04-f192-431c-baab-1352b1d7fcc9-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:31:42.194670 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.194675    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde82b04-f192-431c-baab-1352b1d7fcc9-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:31:42.194861 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.194684    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde82b04-f192-431c-baab-1352b1d7fcc9-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:31:42.194861 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.194691    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde82b04-f192-431c-baab-1352b1d7fcc9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:31:42.396353 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.396324    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"]
Sep 17 23:31:42.400284 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:42.400259    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-tbcm9"]
Sep 17 23:31:43.500294 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:43.500260    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde82b04-f192-431c-baab-1352b1d7fcc9" path="/var/lib/kubelet/pods/cde82b04-f192-431c-baab-1352b1d7fcc9/volumes"
Sep 17 23:31:45.745237 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.745206    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"]
Sep 17 23:31:45.745646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.745475    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerName="storage-initializer"
Sep 17 23:31:45.745646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.745489    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerName="storage-initializer"
Sep 17 23:31:45.745646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.745500    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerName="kube-rbac-proxy"
Sep 17 23:31:45.745646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.745507    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerName="kube-rbac-proxy"
Sep 17 23:31:45.745646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.745518    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerName="kserve-container"
Sep 17 23:31:45.745646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.745523    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerName="kserve-container"
Sep 17 23:31:45.745646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.745566    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerName="kserve-container"
Sep 17 23:31:45.745646 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.745576    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde82b04-f192-431c-baab-1352b1d7fcc9" containerName="kube-rbac-proxy"
Sep 17 23:31:45.750453 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.750427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.753159 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.753095    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:31:45.753294 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.753095    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 23:31:45.753294 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.753108    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:31:45.753294 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.753241    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:31:45.753294 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.753106    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 23:31:45.753464 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.753111    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:31:45.754031 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.754018    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:31:45.757767 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.757751    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"]
Sep 17 23:31:45.820038 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.820002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71f1d0c8-31be-4422-8f02-6806db7b091b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.820038 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.820035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8vbc\" (UniqueName: \"kubernetes.io/projected/71f1d0c8-31be-4422-8f02-6806db7b091b-kube-api-access-b8vbc\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.820190 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.820083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71f1d0c8-31be-4422-8f02-6806db7b091b-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.820190 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.820105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71f1d0c8-31be-4422-8f02-6806db7b091b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.820190 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.820141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71f1d0c8-31be-4422-8f02-6806db7b091b-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.920619 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.920592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71f1d0c8-31be-4422-8f02-6806db7b091b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.920778 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.920675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71f1d0c8-31be-4422-8f02-6806db7b091b-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.920778 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.920706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71f1d0c8-31be-4422-8f02-6806db7b091b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.920778 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.920741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8vbc\" (UniqueName: \"kubernetes.io/projected/71f1d0c8-31be-4422-8f02-6806db7b091b-kube-api-access-b8vbc\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.920778 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:31:45.920750    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 23:31:45.920973 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.920796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71f1d0c8-31be-4422-8f02-6806db7b091b-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.920973 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:31:45.920820    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f1d0c8-31be-4422-8f02-6806db7b091b-proxy-tls podName:71f1d0c8-31be-4422-8f02-6806db7b091b nodeName:}" failed. No retries permitted until 2026-09-17 23:31:46.42080325 +0000 UTC m=+1239.465179387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71f1d0c8-31be-4422-8f02-6806db7b091b-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" (UID: "71f1d0c8-31be-4422-8f02-6806db7b091b") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 23:31:45.921201 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.921173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71f1d0c8-31be-4422-8f02-6806db7b091b-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.921344 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.921327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71f1d0c8-31be-4422-8f02-6806db7b091b-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.921385 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.921353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71f1d0c8-31be-4422-8f02-6806db7b091b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:45.929427 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:45.929406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8vbc\" (UniqueName: \"kubernetes.io/projected/71f1d0c8-31be-4422-8f02-6806db7b091b-kube-api-access-b8vbc\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:46.424567 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:46.424522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71f1d0c8-31be-4422-8f02-6806db7b091b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:46.427067 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:46.427043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71f1d0c8-31be-4422-8f02-6806db7b091b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:46.660561 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:46.660519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:31:46.932773 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:46.930561    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"]
Sep 17 23:31:46.933196 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:31:46.933175    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f1d0c8_31be_4422_8f02_6806db7b091b.slice/crio-5ae5c9cda302f0a1c5ce7062d0fc15383197df938ee59568cf84deb2012bf6e4 WatchSource:0}: Error finding container 5ae5c9cda302f0a1c5ce7062d0fc15383197df938ee59568cf84deb2012bf6e4: Status 404 returned error can't find the container with id 5ae5c9cda302f0a1c5ce7062d0fc15383197df938ee59568cf84deb2012bf6e4
Sep 17 23:31:47.094182 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:47.094148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" event={"ID":"71f1d0c8-31be-4422-8f02-6806db7b091b","Type":"ContainerStarted","Data":"4fc923714a5d02c56c3e98acdaabad6b9713913480e6abd01f346ca05a0d655f"}
Sep 17 23:31:47.094339 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:31:47.094202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" event={"ID":"71f1d0c8-31be-4422-8f02-6806db7b091b","Type":"ContainerStarted","Data":"5ae5c9cda302f0a1c5ce7062d0fc15383197df938ee59568cf84deb2012bf6e4"}
Sep 17 23:32:17.200202 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:32:17.200165    2580 generic.go:356] "Generic (PLEG): container finished" podID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerID="4fc923714a5d02c56c3e98acdaabad6b9713913480e6abd01f346ca05a0d655f" exitCode=0
Sep 17 23:32:17.200676 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:32:17.200213    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" event={"ID":"71f1d0c8-31be-4422-8f02-6806db7b091b","Type":"ContainerDied","Data":"4fc923714a5d02c56c3e98acdaabad6b9713913480e6abd01f346ca05a0d655f"}
Sep 17 23:32:18.206581 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:32:18.206545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" event={"ID":"71f1d0c8-31be-4422-8f02-6806db7b091b","Type":"ContainerStarted","Data":"8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62"}
Sep 17 23:32:18.206962 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:32:18.206647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" event={"ID":"71f1d0c8-31be-4422-8f02-6806db7b091b","Type":"ContainerStarted","Data":"d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d"}
Sep 17 23:32:18.207017 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:32:18.206957    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:32:18.207017 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:32:18.207005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:32:18.236254 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:32:18.236176    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" podStartSLOduration=3.997471356 podStartE2EDuration="33.236154718s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.238683362s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:18.233472676 +0000 UTC m=+1271.277848837" watchObservedRunningTime="2026-09-17 23:32:18.236154718 +0000 UTC m=+1271.280530878"
Sep 17 23:32:24.217898 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:32:24.217866    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:33:07.485418 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:33:07.485381    2580 scope.go:117] "RemoveContainer" containerID="6c67e2cf76131343de3a813d98f055c9b6964daf66fbd3cda6590bed7053a0e4"
Sep 17 23:33:48.529925 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:33:48.529841    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:33:59.537802 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:33:59.537758    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.22:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 23:34:02.713043 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:02.713009    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"]
Sep 17 23:34:02.713512 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:02.713312    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerName="kserve-container" containerID="cri-o://d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d" gracePeriod=30
Sep 17 23:34:02.713512 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:02.713377    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerName="kube-rbac-proxy" containerID="cri-o://8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62" gracePeriod=30
Sep 17 23:34:03.585332 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:03.585292    2580 generic.go:356] "Generic (PLEG): container finished" podID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerID="8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62" exitCode=2
Sep 17 23:34:03.585532 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:03.585371    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" event={"ID":"71f1d0c8-31be-4422-8f02-6806db7b091b","Type":"ContainerDied","Data":"8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62"}
Sep 17 23:34:04.311092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.311066    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:34:04.452006 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.451927    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71f1d0c8-31be-4422-8f02-6806db7b091b-cabundle-cert\") pod \"71f1d0c8-31be-4422-8f02-6806db7b091b\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") "
Sep 17 23:34:04.452006 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.451969    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8vbc\" (UniqueName: \"kubernetes.io/projected/71f1d0c8-31be-4422-8f02-6806db7b091b-kube-api-access-b8vbc\") pod \"71f1d0c8-31be-4422-8f02-6806db7b091b\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") "
Sep 17 23:34:04.452006 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.452007    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71f1d0c8-31be-4422-8f02-6806db7b091b-kserve-provision-location\") pod \"71f1d0c8-31be-4422-8f02-6806db7b091b\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") "
Sep 17 23:34:04.452264 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.452144    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71f1d0c8-31be-4422-8f02-6806db7b091b-kube-rbac-proxy-sar-config\") pod \"71f1d0c8-31be-4422-8f02-6806db7b091b\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") "
Sep 17 23:34:04.452264 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.452249    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71f1d0c8-31be-4422-8f02-6806db7b091b-proxy-tls\") pod \"71f1d0c8-31be-4422-8f02-6806db7b091b\" (UID: \"71f1d0c8-31be-4422-8f02-6806db7b091b\") "
Sep 17 23:34:04.452352 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.452311    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f1d0c8-31be-4422-8f02-6806db7b091b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71f1d0c8-31be-4422-8f02-6806db7b091b" (UID: "71f1d0c8-31be-4422-8f02-6806db7b091b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:34:04.452480 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.452461    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71f1d0c8-31be-4422-8f02-6806db7b091b-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:34:04.452550 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.452475    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f1d0c8-31be-4422-8f02-6806db7b091b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71f1d0c8-31be-4422-8f02-6806db7b091b" (UID: "71f1d0c8-31be-4422-8f02-6806db7b091b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:34:04.455650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.454789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f1d0c8-31be-4422-8f02-6806db7b091b-kube-api-access-b8vbc" (OuterVolumeSpecName: "kube-api-access-b8vbc") pod "71f1d0c8-31be-4422-8f02-6806db7b091b" (UID: "71f1d0c8-31be-4422-8f02-6806db7b091b"). InnerVolumeSpecName "kube-api-access-b8vbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:34:04.456760 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.456717    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f1d0c8-31be-4422-8f02-6806db7b091b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71f1d0c8-31be-4422-8f02-6806db7b091b" (UID: "71f1d0c8-31be-4422-8f02-6806db7b091b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:34:04.459699 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.459586    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f1d0c8-31be-4422-8f02-6806db7b091b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71f1d0c8-31be-4422-8f02-6806db7b091b" (UID: "71f1d0c8-31be-4422-8f02-6806db7b091b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:34:04.553713 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.553670    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71f1d0c8-31be-4422-8f02-6806db7b091b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:34:04.553713 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.553711    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71f1d0c8-31be-4422-8f02-6806db7b091b-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:34:04.553977 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.553733    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8vbc\" (UniqueName: \"kubernetes.io/projected/71f1d0c8-31be-4422-8f02-6806db7b091b-kube-api-access-b8vbc\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:34:04.553977 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.553751    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71f1d0c8-31be-4422-8f02-6806db7b091b-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:34:04.597969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.593783    2580 generic.go:356] "Generic (PLEG): container finished" podID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerID="d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d" exitCode=0
Sep 17 23:34:04.597969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.593835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" event={"ID":"71f1d0c8-31be-4422-8f02-6806db7b091b","Type":"ContainerDied","Data":"d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d"}
Sep 17 23:34:04.597969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.593898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" event={"ID":"71f1d0c8-31be-4422-8f02-6806db7b091b","Type":"ContainerDied","Data":"5ae5c9cda302f0a1c5ce7062d0fc15383197df938ee59568cf84deb2012bf6e4"}
Sep 17 23:34:04.597969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.593938    2580 scope.go:117] "RemoveContainer" containerID="8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62"
Sep 17 23:34:04.597969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.594133    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"
Sep 17 23:34:04.611858 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.611831    2580 scope.go:117] "RemoveContainer" containerID="d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d"
Sep 17 23:34:04.619870 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.619852    2580 scope.go:117] "RemoveContainer" containerID="4fc923714a5d02c56c3e98acdaabad6b9713913480e6abd01f346ca05a0d655f"
Sep 17 23:34:04.625560 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.625537    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"]
Sep 17 23:34:04.628382 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.628358    2580 scope.go:117] "RemoveContainer" containerID="8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62"
Sep 17 23:34:04.628810 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:34:04.628757    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62\": container with ID starting with 8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62 not found: ID does not exist" containerID="8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62"
Sep 17 23:34:04.628939 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.628820    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62"} err="failed to get container status \"8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62\": rpc error: code = NotFound desc = could not find container \"8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62\": container with ID starting with 8a31bef83b0bc47e223af1e4764a2f72ac2863b626efb7d1f945cf32aaea1c62 not found: ID does not exist"
Sep 17 23:34:04.628939 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.628843    2580 scope.go:117] "RemoveContainer" containerID="d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d"
Sep 17 23:34:04.629430 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:34:04.629410    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d\": container with ID starting with d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d not found: ID does not exist" containerID="d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d"
Sep 17 23:34:04.629521 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.629438    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d"} err="failed to get container status \"d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d\": rpc error: code = NotFound desc = could not find container \"d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d\": container with ID starting with d3a224faf8e824a29f64408a863c9aaa6b51ef75f9ac050f6a494118afb5690d not found: ID does not exist"
Sep 17 23:34:04.629521 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.629461    2580 scope.go:117] "RemoveContainer" containerID="4fc923714a5d02c56c3e98acdaabad6b9713913480e6abd01f346ca05a0d655f"
Sep 17 23:34:04.629792 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:34:04.629770    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc923714a5d02c56c3e98acdaabad6b9713913480e6abd01f346ca05a0d655f\": container with ID starting with 4fc923714a5d02c56c3e98acdaabad6b9713913480e6abd01f346ca05a0d655f not found: ID does not exist" containerID="4fc923714a5d02c56c3e98acdaabad6b9713913480e6abd01f346ca05a0d655f"
Sep 17 23:34:04.629863 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.629794    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc923714a5d02c56c3e98acdaabad6b9713913480e6abd01f346ca05a0d655f"} err="failed to get container status \"4fc923714a5d02c56c3e98acdaabad6b9713913480e6abd01f346ca05a0d655f\": rpc error: code = NotFound desc = could not find container \"4fc923714a5d02c56c3e98acdaabad6b9713913480e6abd01f346ca05a0d655f\": container with ID starting with 4fc923714a5d02c56c3e98acdaabad6b9713913480e6abd01f346ca05a0d655f not found: ID does not exist"
Sep 17 23:34:04.630388 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:04.630372    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj"]
Sep 17 23:34:05.213068 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:05.213025    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9ftbj" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 23:34:05.503607 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:05.503513    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" path="/var/lib/kubelet/pods/71f1d0c8-31be-4422-8f02-6806db7b091b/volumes"
Sep 17 23:34:07.923591 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.923553    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"]
Sep 17 23:34:07.924045 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.923982    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerName="storage-initializer"
Sep 17 23:34:07.924045 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.924008    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerName="storage-initializer"
Sep 17 23:34:07.924045 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.924039    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerName="kube-rbac-proxy"
Sep 17 23:34:07.924045 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.924048    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerName="kube-rbac-proxy"
Sep 17 23:34:07.924268 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.924061    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerName="kserve-container"
Sep 17 23:34:07.924268 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.924071    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerName="kserve-container"
Sep 17 23:34:07.924268 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.924144    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerName="kserve-container"
Sep 17 23:34:07.924268 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.924155    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f1d0c8-31be-4422-8f02-6806db7b091b" containerName="kube-rbac-proxy"
Sep 17 23:34:07.928380 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.928359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:07.931009 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.930984    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:34:07.931009 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.930999    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:34:07.931190 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.931000    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 17 23:34:07.931190 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.931093    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:34:07.931293 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.931270    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 17 23:34:07.931916 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.931899    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:34:07.932010 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.931905    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:34:07.937835 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.937814    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"]
Sep 17 23:34:07.982976 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.982947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbkhz\" (UniqueName: \"kubernetes.io/projected/fd117893-9ee7-4e5f-b02c-a0e456171557-kube-api-access-zbkhz\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:07.982976 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.982981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd117893-9ee7-4e5f-b02c-a0e456171557-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:07.983175 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.983019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd117893-9ee7-4e5f-b02c-a0e456171557-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:07.983175 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.983085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd117893-9ee7-4e5f-b02c-a0e456171557-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:07.983175 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:07.983128    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd117893-9ee7-4e5f-b02c-a0e456171557-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:08.083419 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:08.083381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbkhz\" (UniqueName: \"kubernetes.io/projected/fd117893-9ee7-4e5f-b02c-a0e456171557-kube-api-access-zbkhz\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:08.083419 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:08.083421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd117893-9ee7-4e5f-b02c-a0e456171557-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:08.083641 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:08.083445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd117893-9ee7-4e5f-b02c-a0e456171557-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:08.083641 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:08.083466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd117893-9ee7-4e5f-b02c-a0e456171557-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:08.083641 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:08.083495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd117893-9ee7-4e5f-b02c-a0e456171557-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:08.083641 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:34:08.083575    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 17 23:34:08.083890 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:34:08.083712    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd117893-9ee7-4e5f-b02c-a0e456171557-proxy-tls podName:fd117893-9ee7-4e5f-b02c-a0e456171557 nodeName:}" failed. No retries permitted until 2026-09-17 23:34:08.583687894 +0000 UTC m=+1381.628064046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fd117893-9ee7-4e5f-b02c-a0e456171557-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" (UID: "fd117893-9ee7-4e5f-b02c-a0e456171557") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 17 23:34:08.083890 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:08.083865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd117893-9ee7-4e5f-b02c-a0e456171557-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:08.084270 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:08.084249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd117893-9ee7-4e5f-b02c-a0e456171557-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:08.084309 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:08.084249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd117893-9ee7-4e5f-b02c-a0e456171557-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:08.092140 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:08.092115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbkhz\" (UniqueName: \"kubernetes.io/projected/fd117893-9ee7-4e5f-b02c-a0e456171557-kube-api-access-zbkhz\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:08.586825 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:08.586776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd117893-9ee7-4e5f-b02c-a0e456171557-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:08.589410 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:08.589378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd117893-9ee7-4e5f-b02c-a0e456171557-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:08.838546 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:08.838467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:09.111562 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:09.111490    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"]
Sep 17 23:34:09.114944 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:34:09.114917    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd117893_9ee7_4e5f_b02c_a0e456171557.slice/crio-af09bb57104c68c1092cdd3bbd041729b853d34198ea6ce0843067bb173dc281 WatchSource:0}: Error finding container af09bb57104c68c1092cdd3bbd041729b853d34198ea6ce0843067bb173dc281: Status 404 returned error can't find the container with id af09bb57104c68c1092cdd3bbd041729b853d34198ea6ce0843067bb173dc281
Sep 17 23:34:09.614812 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:09.614776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" event={"ID":"fd117893-9ee7-4e5f-b02c-a0e456171557","Type":"ContainerStarted","Data":"73a18d387dbd4a1655c3c2c4f95b811a39df006d1cacb22daca2fb31b7a443d7"}
Sep 17 23:34:09.615015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:09.614839    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" event={"ID":"fd117893-9ee7-4e5f-b02c-a0e456171557","Type":"ContainerStarted","Data":"af09bb57104c68c1092cdd3bbd041729b853d34198ea6ce0843067bb173dc281"}
Sep 17 23:34:14.638487 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:14.638453    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerID="73a18d387dbd4a1655c3c2c4f95b811a39df006d1cacb22daca2fb31b7a443d7" exitCode=0
Sep 17 23:34:14.638990 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:14.638510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" event={"ID":"fd117893-9ee7-4e5f-b02c-a0e456171557","Type":"ContainerDied","Data":"73a18d387dbd4a1655c3c2c4f95b811a39df006d1cacb22daca2fb31b7a443d7"}
Sep 17 23:34:15.644714 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:15.644679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" event={"ID":"fd117893-9ee7-4e5f-b02c-a0e456171557","Type":"ContainerStarted","Data":"8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746"}
Sep 17 23:34:15.645216 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:15.644740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" event={"ID":"fd117893-9ee7-4e5f-b02c-a0e456171557","Type":"ContainerStarted","Data":"eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be"}
Sep 17 23:34:15.645216 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:15.644934    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:15.645216 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:15.644978    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:34:15.663336 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:15.663282    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" podStartSLOduration=3.690681574 podStartE2EDuration="8.66326519s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.972583616s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:15.662171106 +0000 UTC m=+1388.706547267" watchObservedRunningTime="2026-09-17 23:34:15.66326519 +0000 UTC m=+1388.707641409"
Sep 17 23:34:21.654768 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:34:21.654740    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:35:50.529758 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:35:50.529721    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:36:01.537804 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:01.537758    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" podUID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.23:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 23:36:03.978273 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:03.978239    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"]
Sep 17 23:36:03.978758 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:03.978539    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" podUID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerName="kserve-container" containerID="cri-o://eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be" gracePeriod=30
Sep 17 23:36:03.978758 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:03.978628    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" podUID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerName="kube-rbac-proxy" containerID="cri-o://8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746" gracePeriod=30
Sep 17 23:36:05.033722 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.033689    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerID="8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746" exitCode=2
Sep 17 23:36:05.034106 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.033766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" event={"ID":"fd117893-9ee7-4e5f-b02c-a0e456171557","Type":"ContainerDied","Data":"8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746"}
Sep 17 23:36:05.414247 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.414225    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:36:05.508647 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.508241    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd117893-9ee7-4e5f-b02c-a0e456171557-kube-rbac-proxy-sar-config\") pod \"fd117893-9ee7-4e5f-b02c-a0e456171557\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") "
Sep 17 23:36:05.508647 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.508322    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbkhz\" (UniqueName: \"kubernetes.io/projected/fd117893-9ee7-4e5f-b02c-a0e456171557-kube-api-access-zbkhz\") pod \"fd117893-9ee7-4e5f-b02c-a0e456171557\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") "
Sep 17 23:36:05.508647 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.508365    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd117893-9ee7-4e5f-b02c-a0e456171557-kserve-provision-location\") pod \"fd117893-9ee7-4e5f-b02c-a0e456171557\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") "
Sep 17 23:36:05.508647 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.508412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd117893-9ee7-4e5f-b02c-a0e456171557-cabundle-cert\") pod \"fd117893-9ee7-4e5f-b02c-a0e456171557\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") "
Sep 17 23:36:05.508647 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.508457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd117893-9ee7-4e5f-b02c-a0e456171557-proxy-tls\") pod \"fd117893-9ee7-4e5f-b02c-a0e456171557\" (UID: \"fd117893-9ee7-4e5f-b02c-a0e456171557\") "
Sep 17 23:36:05.510562 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.510284    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd117893-9ee7-4e5f-b02c-a0e456171557-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd117893-9ee7-4e5f-b02c-a0e456171557" (UID: "fd117893-9ee7-4e5f-b02c-a0e456171557"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:36:05.512976 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.512936    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd117893-9ee7-4e5f-b02c-a0e456171557-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fd117893-9ee7-4e5f-b02c-a0e456171557" (UID: "fd117893-9ee7-4e5f-b02c-a0e456171557"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:36:05.513849 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.513814    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd117893-9ee7-4e5f-b02c-a0e456171557-kube-api-access-zbkhz" (OuterVolumeSpecName: "kube-api-access-zbkhz") pod "fd117893-9ee7-4e5f-b02c-a0e456171557" (UID: "fd117893-9ee7-4e5f-b02c-a0e456171557"). InnerVolumeSpecName "kube-api-access-zbkhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:36:05.514823 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.514802    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd117893-9ee7-4e5f-b02c-a0e456171557-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd117893-9ee7-4e5f-b02c-a0e456171557" (UID: "fd117893-9ee7-4e5f-b02c-a0e456171557"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:36:05.519564 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.519530    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd117893-9ee7-4e5f-b02c-a0e456171557-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd117893-9ee7-4e5f-b02c-a0e456171557" (UID: "fd117893-9ee7-4e5f-b02c-a0e456171557"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:36:05.609915 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.609839    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd117893-9ee7-4e5f-b02c-a0e456171557-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:36:05.609915 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.609864    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbkhz\" (UniqueName: \"kubernetes.io/projected/fd117893-9ee7-4e5f-b02c-a0e456171557-kube-api-access-zbkhz\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:36:05.609915 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.609873    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd117893-9ee7-4e5f-b02c-a0e456171557-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:36:05.609915 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.609883    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd117893-9ee7-4e5f-b02c-a0e456171557-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:36:05.609915 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:05.609893    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd117893-9ee7-4e5f-b02c-a0e456171557-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:36:06.039414 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.039385    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerID="eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be" exitCode=0
Sep 17 23:36:06.039810 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.039443    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"
Sep 17 23:36:06.039810 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.039467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" event={"ID":"fd117893-9ee7-4e5f-b02c-a0e456171557","Type":"ContainerDied","Data":"eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be"}
Sep 17 23:36:06.039810 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.039547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq" event={"ID":"fd117893-9ee7-4e5f-b02c-a0e456171557","Type":"ContainerDied","Data":"af09bb57104c68c1092cdd3bbd041729b853d34198ea6ce0843067bb173dc281"}
Sep 17 23:36:06.039810 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.039573    2580 scope.go:117] "RemoveContainer" containerID="8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746"
Sep 17 23:36:06.048244 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.048226    2580 scope.go:117] "RemoveContainer" containerID="eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be"
Sep 17 23:36:06.056290 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.056270    2580 scope.go:117] "RemoveContainer" containerID="73a18d387dbd4a1655c3c2c4f95b811a39df006d1cacb22daca2fb31b7a443d7"
Sep 17 23:36:06.061510 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.061487    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"]
Sep 17 23:36:06.064585 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.064559    2580 scope.go:117] "RemoveContainer" containerID="8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746"
Sep 17 23:36:06.064964 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:36:06.064944    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746\": container with ID starting with 8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746 not found: ID does not exist" containerID="8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746"
Sep 17 23:36:06.065030 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.064974    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746"} err="failed to get container status \"8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746\": rpc error: code = NotFound desc = could not find container \"8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746\": container with ID starting with 8d83b83d363c402aef6c7fa462da96faa37a658669f37e0be1d80d0803a53746 not found: ID does not exist"
Sep 17 23:36:06.065030 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.064992    2580 scope.go:117] "RemoveContainer" containerID="eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be"
Sep 17 23:36:06.065283 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:36:06.065265    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be\": container with ID starting with eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be not found: ID does not exist" containerID="eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be"
Sep 17 23:36:06.065331 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.065288    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be"} err="failed to get container status \"eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be\": rpc error: code = NotFound desc = could not find container \"eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be\": container with ID starting with eaa0841395ae5e10d6bba29182c6f16f490923874a9ad9d72846612e4103e7be not found: ID does not exist"
Sep 17 23:36:06.065331 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.065303    2580 scope.go:117] "RemoveContainer" containerID="73a18d387dbd4a1655c3c2c4f95b811a39df006d1cacb22daca2fb31b7a443d7"
Sep 17 23:36:06.065543 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:36:06.065529    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a18d387dbd4a1655c3c2c4f95b811a39df006d1cacb22daca2fb31b7a443d7\": container with ID starting with 73a18d387dbd4a1655c3c2c4f95b811a39df006d1cacb22daca2fb31b7a443d7 not found: ID does not exist" containerID="73a18d387dbd4a1655c3c2c4f95b811a39df006d1cacb22daca2fb31b7a443d7"
Sep 17 23:36:06.065584 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.065545    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a18d387dbd4a1655c3c2c4f95b811a39df006d1cacb22daca2fb31b7a443d7"} err="failed to get container status \"73a18d387dbd4a1655c3c2c4f95b811a39df006d1cacb22daca2fb31b7a443d7\": rpc error: code = NotFound desc = could not find container \"73a18d387dbd4a1655c3c2c4f95b811a39df006d1cacb22daca2fb31b7a443d7\": container with ID starting with 73a18d387dbd4a1655c3c2c4f95b811a39df006d1cacb22daca2fb31b7a443d7 not found: ID does not exist"
Sep 17 23:36:06.065696 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:06.065678    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-zcccq"]
Sep 17 23:36:07.500599 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:07.500558    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd117893-9ee7-4e5f-b02c-a0e456171557" path="/var/lib/kubelet/pods/fd117893-9ee7-4e5f-b02c-a0e456171557/volumes"
Sep 17 23:36:07.514976 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:07.514952    2580 scope.go:117] "RemoveContainer" containerID="7fe1c546dd5dd8e5d0ec0505af1c021993e53f7bbaeb998de7cae4d0e15d062c"
Sep 17 23:36:07.522095 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:07.522077    2580 scope.go:117] "RemoveContainer" containerID="c7b5170794aff6cd036eb83f67d0411cded5c9b189cea64a1dd2d5cef29fa6bc"
Sep 17 23:36:09.198495 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.198456    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"]
Sep 17 23:36:09.198978 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.198816    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerName="kserve-container"
Sep 17 23:36:09.198978 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.198835    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerName="kserve-container"
Sep 17 23:36:09.198978 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.198849    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerName="storage-initializer"
Sep 17 23:36:09.198978 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.198858    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerName="storage-initializer"
Sep 17 23:36:09.198978 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.198877    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerName="kube-rbac-proxy"
Sep 17 23:36:09.198978 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.198885    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerName="kube-rbac-proxy"
Sep 17 23:36:09.198978 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.198956    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerName="kube-rbac-proxy"
Sep 17 23:36:09.198978 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.198965    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd117893-9ee7-4e5f-b02c-a0e456171557" containerName="kserve-container"
Sep 17 23:36:09.203532 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.203510    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.205977 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.205950    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:36:09.206095 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.205982    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:36:09.206095 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.205955    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 17 23:36:09.206298 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.206283    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:36:09.206919 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.206905    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 17 23:36:09.206988 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.206929    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:36:09.206988 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.206929    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:36:09.210909 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.210889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"]
Sep 17 23:36:09.232387 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.232355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4d39167-6750-435b-9ac4-943929640618-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.232538 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.232398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4d39167-6750-435b-9ac4-943929640618-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.232538 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.232424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4d39167-6750-435b-9ac4-943929640618-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.232538 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.232481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4d39167-6750-435b-9ac4-943929640618-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.232538 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.232528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbsc\" (UniqueName: \"kubernetes.io/projected/a4d39167-6750-435b-9ac4-943929640618-kube-api-access-tbbsc\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.333708 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.333677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbsc\" (UniqueName: \"kubernetes.io/projected/a4d39167-6750-435b-9ac4-943929640618-kube-api-access-tbbsc\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.333833 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.333719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4d39167-6750-435b-9ac4-943929640618-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.333833 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.333745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4d39167-6750-435b-9ac4-943929640618-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.333833 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.333776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4d39167-6750-435b-9ac4-943929640618-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.333833 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.333797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4d39167-6750-435b-9ac4-943929640618-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.334031 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:36:09.333890    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 17 23:36:09.334031 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:36:09.333972    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4d39167-6750-435b-9ac4-943929640618-proxy-tls podName:a4d39167-6750-435b-9ac4-943929640618 nodeName:}" failed. No retries permitted until 2026-09-17 23:36:09.833950348 +0000 UTC m=+1502.878326489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a4d39167-6750-435b-9ac4-943929640618-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" (UID: "a4d39167-6750-435b-9ac4-943929640618") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 17 23:36:09.334149 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.334129    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4d39167-6750-435b-9ac4-943929640618-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.334435 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.334418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4d39167-6750-435b-9ac4-943929640618-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.334435 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.334427    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4d39167-6750-435b-9ac4-943929640618-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.342796 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.342771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbsc\" (UniqueName: \"kubernetes.io/projected/a4d39167-6750-435b-9ac4-943929640618-kube-api-access-tbbsc\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.837517 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.837476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4d39167-6750-435b-9ac4-943929640618-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:09.840102 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:09.840079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4d39167-6750-435b-9ac4-943929640618-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:10.114293 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:10.114207    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:10.382723 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:10.382644    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"]
Sep 17 23:36:10.385830 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:36:10.385807    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d39167_6750_435b_9ac4_943929640618.slice/crio-9a99a210ef9f255b19d6dd8e9afed18e9791d79be2d4c4df3d79b6c9b049f388 WatchSource:0}: Error finding container 9a99a210ef9f255b19d6dd8e9afed18e9791d79be2d4c4df3d79b6c9b049f388: Status 404 returned error can't find the container with id 9a99a210ef9f255b19d6dd8e9afed18e9791d79be2d4c4df3d79b6c9b049f388
Sep 17 23:36:11.059640 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:11.059569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" event={"ID":"a4d39167-6750-435b-9ac4-943929640618","Type":"ContainerStarted","Data":"09d484ae411f748b6dff54d197971908f9bdcc1b2da1585aa8a012263a50e3a4"}
Sep 17 23:36:11.059911 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:11.059676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" event={"ID":"a4d39167-6750-435b-9ac4-943929640618","Type":"ContainerStarted","Data":"9a99a210ef9f255b19d6dd8e9afed18e9791d79be2d4c4df3d79b6c9b049f388"}
Sep 17 23:36:20.091376 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:20.091344    2580 generic.go:356] "Generic (PLEG): container finished" podID="a4d39167-6750-435b-9ac4-943929640618" containerID="09d484ae411f748b6dff54d197971908f9bdcc1b2da1585aa8a012263a50e3a4" exitCode=0
Sep 17 23:36:20.091747 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:20.091408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" event={"ID":"a4d39167-6750-435b-9ac4-943929640618","Type":"ContainerDied","Data":"09d484ae411f748b6dff54d197971908f9bdcc1b2da1585aa8a012263a50e3a4"}
Sep 17 23:36:21.102811 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:21.102776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" event={"ID":"a4d39167-6750-435b-9ac4-943929640618","Type":"ContainerStarted","Data":"e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5"}
Sep 17 23:36:21.103308 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:21.102857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" event={"ID":"a4d39167-6750-435b-9ac4-943929640618","Type":"ContainerStarted","Data":"dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3"}
Sep 17 23:36:21.103308 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:21.103180    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:21.103308 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:21.103232    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:21.104893 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:21.104847    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 23:36:21.120112 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:21.119760    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" podStartSLOduration=2.494240667 podStartE2EDuration="12.119744533s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.625503866s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:21.119413207 +0000 UTC m=+1514.163789366" watchObservedRunningTime="2026-09-17 23:36:21.119744533 +0000 UTC m=+1514.164120697"
Sep 17 23:36:22.107652 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:22.107599    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 23:36:27.112791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:27.112761    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:27.113369 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:27.113343    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 23:36:37.114033 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:37.113948    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:51.235165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:51.235132    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"]
Sep 17 23:36:51.237715 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:51.235398    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="kserve-container" containerID="cri-o://dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3" gracePeriod=30
Sep 17 23:36:51.237715 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:51.235479    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="kube-rbac-proxy" containerID="cri-o://e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5" gracePeriod=30
Sep 17 23:36:52.108489 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:52.108444    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 23:36:52.217223 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:52.217188    2580 generic.go:356] "Generic (PLEG): container finished" podID="a4d39167-6750-435b-9ac4-943929640618" containerID="e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5" exitCode=2
Sep 17 23:36:52.217393 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:52.217266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" event={"ID":"a4d39167-6750-435b-9ac4-943929640618","Type":"ContainerDied","Data":"e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5"}
Sep 17 23:36:52.977102 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:52.977073    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:53.052540 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.052519    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbbsc\" (UniqueName: \"kubernetes.io/projected/a4d39167-6750-435b-9ac4-943929640618-kube-api-access-tbbsc\") pod \"a4d39167-6750-435b-9ac4-943929640618\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") "
Sep 17 23:36:53.052817 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.052567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4d39167-6750-435b-9ac4-943929640618-cabundle-cert\") pod \"a4d39167-6750-435b-9ac4-943929640618\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") "
Sep 17 23:36:53.052817 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.052583    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4d39167-6750-435b-9ac4-943929640618-kube-rbac-proxy-sar-config\") pod \"a4d39167-6750-435b-9ac4-943929640618\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") "
Sep 17 23:36:53.052817 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.052607    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4d39167-6750-435b-9ac4-943929640618-kserve-provision-location\") pod \"a4d39167-6750-435b-9ac4-943929640618\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") "
Sep 17 23:36:53.052817 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.052689    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4d39167-6750-435b-9ac4-943929640618-proxy-tls\") pod \"a4d39167-6750-435b-9ac4-943929640618\" (UID: \"a4d39167-6750-435b-9ac4-943929640618\") "
Sep 17 23:36:53.053017 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.052971    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d39167-6750-435b-9ac4-943929640618-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4d39167-6750-435b-9ac4-943929640618" (UID: "a4d39167-6750-435b-9ac4-943929640618"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:36:53.053017 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.052975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d39167-6750-435b-9ac4-943929640618-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4d39167-6750-435b-9ac4-943929640618" (UID: "a4d39167-6750-435b-9ac4-943929640618"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:36:53.053505 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.053488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d39167-6750-435b-9ac4-943929640618-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4d39167-6750-435b-9ac4-943929640618" (UID: "a4d39167-6750-435b-9ac4-943929640618"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:36:53.054722 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.054707    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d39167-6750-435b-9ac4-943929640618-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4d39167-6750-435b-9ac4-943929640618" (UID: "a4d39167-6750-435b-9ac4-943929640618"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:36:53.054834 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.054812    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d39167-6750-435b-9ac4-943929640618-kube-api-access-tbbsc" (OuterVolumeSpecName: "kube-api-access-tbbsc") pod "a4d39167-6750-435b-9ac4-943929640618" (UID: "a4d39167-6750-435b-9ac4-943929640618"). InnerVolumeSpecName "kube-api-access-tbbsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:36:53.153415 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.153374    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4d39167-6750-435b-9ac4-943929640618-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:36:53.153415 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.153402    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4d39167-6750-435b-9ac4-943929640618-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:36:53.153415 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.153412    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4d39167-6750-435b-9ac4-943929640618-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:36:53.153415 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.153421    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4d39167-6750-435b-9ac4-943929640618-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:36:53.153704 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.153429    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbbsc\" (UniqueName: \"kubernetes.io/projected/a4d39167-6750-435b-9ac4-943929640618-kube-api-access-tbbsc\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:36:53.222671 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.222641    2580 generic.go:356] "Generic (PLEG): container finished" podID="a4d39167-6750-435b-9ac4-943929640618" containerID="dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3" exitCode=0
Sep 17 23:36:53.222806 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.222694    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"
Sep 17 23:36:53.222806 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.222694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" event={"ID":"a4d39167-6750-435b-9ac4-943929640618","Type":"ContainerDied","Data":"dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3"}
Sep 17 23:36:53.222806 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.222756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb" event={"ID":"a4d39167-6750-435b-9ac4-943929640618","Type":"ContainerDied","Data":"9a99a210ef9f255b19d6dd8e9afed18e9791d79be2d4c4df3d79b6c9b049f388"}
Sep 17 23:36:53.222806 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.222783    2580 scope.go:117] "RemoveContainer" containerID="e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5"
Sep 17 23:36:53.231254 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.231236    2580 scope.go:117] "RemoveContainer" containerID="dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3"
Sep 17 23:36:53.238302 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.238285    2580 scope.go:117] "RemoveContainer" containerID="09d484ae411f748b6dff54d197971908f9bdcc1b2da1585aa8a012263a50e3a4"
Sep 17 23:36:53.244781 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.244750    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"]
Sep 17 23:36:53.245463 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.245435    2580 scope.go:117] "RemoveContainer" containerID="e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5"
Sep 17 23:36:53.245779 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:36:53.245762    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5\": container with ID starting with e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5 not found: ID does not exist" containerID="e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5"
Sep 17 23:36:53.245837 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.245787    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5"} err="failed to get container status \"e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5\": rpc error: code = NotFound desc = could not find container \"e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5\": container with ID starting with e990588d284f136a8b665a14123cd6ee2aa889bb33a1822a63c1dc4975d81fd5 not found: ID does not exist"
Sep 17 23:36:53.245837 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.245803    2580 scope.go:117] "RemoveContainer" containerID="dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3"
Sep 17 23:36:53.246041 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:36:53.246023    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3\": container with ID starting with dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3 not found: ID does not exist" containerID="dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3"
Sep 17 23:36:53.246080 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.246048    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3"} err="failed to get container status \"dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3\": rpc error: code = NotFound desc = could not find container \"dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3\": container with ID starting with dbc3118f298fc50eca4b2ffc381f64f50c45fe41c5940e54e6811ca70899c4b3 not found: ID does not exist"
Sep 17 23:36:53.246080 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.246063    2580 scope.go:117] "RemoveContainer" containerID="09d484ae411f748b6dff54d197971908f9bdcc1b2da1585aa8a012263a50e3a4"
Sep 17 23:36:53.246268 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:36:53.246253    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d484ae411f748b6dff54d197971908f9bdcc1b2da1585aa8a012263a50e3a4\": container with ID starting with 09d484ae411f748b6dff54d197971908f9bdcc1b2da1585aa8a012263a50e3a4 not found: ID does not exist" containerID="09d484ae411f748b6dff54d197971908f9bdcc1b2da1585aa8a012263a50e3a4"
Sep 17 23:36:53.246309 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.246272    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d484ae411f748b6dff54d197971908f9bdcc1b2da1585aa8a012263a50e3a4"} err="failed to get container status \"09d484ae411f748b6dff54d197971908f9bdcc1b2da1585aa8a012263a50e3a4\": rpc error: code = NotFound desc = could not find container \"09d484ae411f748b6dff54d197971908f9bdcc1b2da1585aa8a012263a50e3a4\": container with ID starting with 09d484ae411f748b6dff54d197971908f9bdcc1b2da1585aa8a012263a50e3a4 not found: ID does not exist"
Sep 17 23:36:53.251757 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.251734    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xlmkb"]
Sep 17 23:36:53.500246 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:53.500170    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4d39167-6750-435b-9ac4-943929640618" path="/var/lib/kubelet/pods/a4d39167-6750-435b-9ac4-943929640618/volumes"
Sep 17 23:36:56.472206 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.472168    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"]
Sep 17 23:36:56.472579 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.472460    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="kserve-container"
Sep 17 23:36:56.472579 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.472472    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="kserve-container"
Sep 17 23:36:56.472579 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.472489    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="storage-initializer"
Sep 17 23:36:56.472579 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.472495    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="storage-initializer"
Sep 17 23:36:56.472579 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.472501    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="kube-rbac-proxy"
Sep 17 23:36:56.472579 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.472507    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="kube-rbac-proxy"
Sep 17 23:36:56.472579 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.472551    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="kube-rbac-proxy"
Sep 17 23:36:56.472579 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.472559    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d39167-6750-435b-9ac4-943929640618" containerName="kserve-container"
Sep 17 23:36:56.477226 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.477205    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.479713 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.479683    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:36:56.479971 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.479952    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:36:56.480054 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.480010    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 17 23:36:56.480108 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.480010    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:36:56.480705 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.480688    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:36:56.480800 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.480701    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:36:56.480800 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.480691    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 17 23:36:56.485561 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.485533    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"]
Sep 17 23:36:56.580029 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.579991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.580233 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.580037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.580233 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.580179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xngnv\" (UniqueName: \"kubernetes.io/projected/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kube-api-access-xngnv\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.580346 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.580230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.580346 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.580271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.680852 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.680827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xngnv\" (UniqueName: \"kubernetes.io/projected/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kube-api-access-xngnv\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.680951 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.680861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.680951 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.680880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.680951 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.680908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.680951 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.680929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.681127 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:36:56.681060    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 17 23:36:56.681161 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:36:56.681131    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-proxy-tls podName:cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c nodeName:}" failed. No retries permitted until 2026-09-17 23:36:57.181111662 +0000 UTC m=+1550.225487801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" (UID: "cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 17 23:36:56.681295 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.681266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.681550 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.681533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.681596 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.681557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:56.690455 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:56.690431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xngnv\" (UniqueName: \"kubernetes.io/projected/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kube-api-access-xngnv\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:57.185713 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:57.185677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:57.188189 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:57.188171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:57.386711 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:57.386675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:36:57.655732 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:57.655549    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"]
Sep 17 23:36:57.658300 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:36:57.658267    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadeea98_c6c4_4a13_9a4a_1d68ffb54d1c.slice/crio-cd2c7380986c1c94ae2e7d16e041806924be4688da12a7a28486f3994450cc2a WatchSource:0}: Error finding container cd2c7380986c1c94ae2e7d16e041806924be4688da12a7a28486f3994450cc2a: Status 404 returned error can't find the container with id cd2c7380986c1c94ae2e7d16e041806924be4688da12a7a28486f3994450cc2a
Sep 17 23:36:58.242501 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:58.242464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" event={"ID":"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c","Type":"ContainerStarted","Data":"f3d2839c580fe26a0b7b79e217b231dc05c2f63c22731c2e1e5f7af09b8f6ed8"}
Sep 17 23:36:58.242691 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:36:58.242519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" event={"ID":"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c","Type":"ContainerStarted","Data":"cd2c7380986c1c94ae2e7d16e041806924be4688da12a7a28486f3994450cc2a"}
Sep 17 23:37:07.275248 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:07.275207    2580 generic.go:356] "Generic (PLEG): container finished" podID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerID="f3d2839c580fe26a0b7b79e217b231dc05c2f63c22731c2e1e5f7af09b8f6ed8" exitCode=0
Sep 17 23:37:07.275680 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:07.275281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" event={"ID":"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c","Type":"ContainerDied","Data":"f3d2839c580fe26a0b7b79e217b231dc05c2f63c22731c2e1e5f7af09b8f6ed8"}
Sep 17 23:37:08.280800 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:08.280770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" event={"ID":"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c","Type":"ContainerStarted","Data":"2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810"}
Sep 17 23:37:08.281253 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:08.280837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" event={"ID":"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c","Type":"ContainerStarted","Data":"1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51"}
Sep 17 23:37:08.281253 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:08.281064    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:37:08.301051 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:08.301006    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" podStartSLOduration=3.291992712 podStartE2EDuration="12.300990569s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.008997857s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:08.299701803 +0000 UTC m=+1561.344077964" watchObservedRunningTime="2026-09-17 23:37:08.300990569 +0000 UTC m=+1561.345366727"
Sep 17 23:37:09.284988 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:09.284954    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:37:09.286105 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:09.286081    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 23:37:10.288165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:10.288125    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 23:37:15.292786 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:15.292758    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:37:15.293237 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:15.293211    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 23:37:25.294807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:25.294776    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:37:31.768862 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:31.768826    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"]
Sep 17 23:37:31.769306 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:31.769083    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="kserve-container" containerID="cri-o://1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51" gracePeriod=30
Sep 17 23:37:31.769306 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:31.769120    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="kube-rbac-proxy" containerID="cri-o://2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810" gracePeriod=30
Sep 17 23:37:32.368303 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:32.368273    2580 generic.go:356] "Generic (PLEG): container finished" podID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerID="2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810" exitCode=2
Sep 17 23:37:32.368469 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:32.368327    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" event={"ID":"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c","Type":"ContainerDied","Data":"2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810"}
Sep 17 23:37:33.204593 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.204570    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:37:33.343824 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.343788    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kserve-provision-location\") pod \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") "
Sep 17 23:37:33.344029 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.343854    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-cabundle-cert\") pod \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") "
Sep 17 23:37:33.344029 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.343890    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-proxy-tls\") pod \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") "
Sep 17 23:37:33.344029 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.343911    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xngnv\" (UniqueName: \"kubernetes.io/projected/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kube-api-access-xngnv\") pod \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") "
Sep 17 23:37:33.344029 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.343959    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kube-rbac-proxy-sar-config\") pod \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\" (UID: \"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c\") "
Sep 17 23:37:33.344255 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.344226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" (UID: "cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:33.344375 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.344350    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" (UID: "cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:33.344939 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.344921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" (UID: "cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:33.346126 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.346106    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kube-api-access-xngnv" (OuterVolumeSpecName: "kube-api-access-xngnv") pod "cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" (UID: "cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c"). InnerVolumeSpecName "kube-api-access-xngnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:33.346323 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.346304    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" (UID: "cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:37:33.373834 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.373803    2580 generic.go:356] "Generic (PLEG): container finished" podID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerID="1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51" exitCode=0
Sep 17 23:37:33.373969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.373863    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"
Sep 17 23:37:33.373969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.373870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" event={"ID":"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c","Type":"ContainerDied","Data":"1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51"}
Sep 17 23:37:33.373969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.373918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7" event={"ID":"cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c","Type":"ContainerDied","Data":"cd2c7380986c1c94ae2e7d16e041806924be4688da12a7a28486f3994450cc2a"}
Sep 17 23:37:33.373969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.373944    2580 scope.go:117] "RemoveContainer" containerID="2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810"
Sep 17 23:37:33.382059 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.382038    2580 scope.go:117] "RemoveContainer" containerID="1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51"
Sep 17 23:37:33.389280 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.389261    2580 scope.go:117] "RemoveContainer" containerID="f3d2839c580fe26a0b7b79e217b231dc05c2f63c22731c2e1e5f7af09b8f6ed8"
Sep 17 23:37:33.395302 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.395279    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"]
Sep 17 23:37:33.396495 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.396481    2580 scope.go:117] "RemoveContainer" containerID="2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810"
Sep 17 23:37:33.396855 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:37:33.396834    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810\": container with ID starting with 2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810 not found: ID does not exist" containerID="2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810"
Sep 17 23:37:33.396929 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.396865    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810"} err="failed to get container status \"2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810\": rpc error: code = NotFound desc = could not find container \"2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810\": container with ID starting with 2ffa66caca31d84762b29388c968ff2b34a64fc55e342b90729d803a3c5b6810 not found: ID does not exist"
Sep 17 23:37:33.396929 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.396884    2580 scope.go:117] "RemoveContainer" containerID="1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51"
Sep 17 23:37:33.397149 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:37:33.397131    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51\": container with ID starting with 1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51 not found: ID does not exist" containerID="1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51"
Sep 17 23:37:33.397192 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.397158    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51"} err="failed to get container status \"1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51\": rpc error: code = NotFound desc = could not find container \"1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51\": container with ID starting with 1e82da314e6e7bc00292ae10c46a697d5baee178712b1c2be2492cd4b7114f51 not found: ID does not exist"
Sep 17 23:37:33.397192 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.397177    2580 scope.go:117] "RemoveContainer" containerID="f3d2839c580fe26a0b7b79e217b231dc05c2f63c22731c2e1e5f7af09b8f6ed8"
Sep 17 23:37:33.397420 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:37:33.397401    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d2839c580fe26a0b7b79e217b231dc05c2f63c22731c2e1e5f7af09b8f6ed8\": container with ID starting with f3d2839c580fe26a0b7b79e217b231dc05c2f63c22731c2e1e5f7af09b8f6ed8 not found: ID does not exist" containerID="f3d2839c580fe26a0b7b79e217b231dc05c2f63c22731c2e1e5f7af09b8f6ed8"
Sep 17 23:37:33.397486 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.397420    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d2839c580fe26a0b7b79e217b231dc05c2f63c22731c2e1e5f7af09b8f6ed8"} err="failed to get container status \"f3d2839c580fe26a0b7b79e217b231dc05c2f63c22731c2e1e5f7af09b8f6ed8\": rpc error: code = NotFound desc = could not find container \"f3d2839c580fe26a0b7b79e217b231dc05c2f63c22731c2e1e5f7af09b8f6ed8\": container with ID starting with f3d2839c580fe26a0b7b79e217b231dc05c2f63c22731c2e1e5f7af09b8f6ed8 not found: ID does not exist"
Sep 17 23:37:33.399642 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.399603    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-tjvx7"]
Sep 17 23:37:33.444858 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.444839    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:37:33.444922 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.444863    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:37:33.444922 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.444872    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xngnv\" (UniqueName: \"kubernetes.io/projected/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kube-api-access-xngnv\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:37:33.444922 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.444881    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:37:33.444922 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.444891    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:37:33.500967 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:33.500937    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" path="/var/lib/kubelet/pods/cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c/volumes"
Sep 17 23:37:36.925098 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.925065    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"]
Sep 17 23:37:36.925447 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.925340    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="kube-rbac-proxy"
Sep 17 23:37:36.925447 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.925353    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="kube-rbac-proxy"
Sep 17 23:37:36.925447 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.925362    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="storage-initializer"
Sep 17 23:37:36.925447 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.925369    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="storage-initializer"
Sep 17 23:37:36.925447 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.925380    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="kserve-container"
Sep 17 23:37:36.925447 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.925386    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="kserve-container"
Sep 17 23:37:36.925447 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.925424    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="kube-rbac-proxy"
Sep 17 23:37:36.925447 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.925431    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cadeea98-c6c4-4a13-9a4a-1d68ffb54d1c" containerName="kserve-container"
Sep 17 23:37:36.930112 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.930094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:36.933372 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.933351    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:37:36.933882 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.933865    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 23:37:36.933981 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.933891    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:37:36.934116 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.934099    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:37:36.934193 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.934117    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:37:36.934193 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.934100    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:37:36.934346 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.934332    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 23:37:36.945260 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.945232    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"]
Sep 17 23:37:36.963442 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.963410    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"]
Sep 17 23:37:36.966628 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.966585    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:36.968039 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.968019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kserve-provision-location\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:36.968137 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.968050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2252\" (UniqueName: \"kubernetes.io/projected/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kube-api-access-h2252\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:36.968137 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.968080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xx2r\" (UniqueName: \"kubernetes.io/projected/0026773a-150e-4942-8dd1-239fa023a131-kube-api-access-4xx2r\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:36.968137 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.968123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0026773a-150e-4942-8dd1-239fa023a131-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:36.968234 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.968147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0026773a-150e-4942-8dd1-239fa023a131-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:36.968234 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.968172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0026773a-150e-4942-8dd1-239fa023a131-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:36.968234 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.968194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:36.968234 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.968226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f611c5eb-e8f3-491a-962a-303ee3dd4d30-cabundle-cert\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:36.968352 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.968254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f611c5eb-e8f3-491a-962a-303ee3dd4d30-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:36.968352 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.968289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0026773a-150e-4942-8dd1-239fa023a131-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:36.969974 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.969958    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 23:37:36.985286 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:36.985259    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"]
Sep 17 23:37:37.069116 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2252\" (UniqueName: \"kubernetes.io/projected/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kube-api-access-h2252\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:37.069116 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xx2r\" (UniqueName: \"kubernetes.io/projected/0026773a-150e-4942-8dd1-239fa023a131-kube-api-access-4xx2r\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:37.069391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0026773a-150e-4942-8dd1-239fa023a131-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:37.069391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0026773a-150e-4942-8dd1-239fa023a131-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:37.069391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0026773a-150e-4942-8dd1-239fa023a131-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:37.069391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:37.069391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f611c5eb-e8f3-491a-962a-303ee3dd4d30-cabundle-cert\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:37.069391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f611c5eb-e8f3-491a-962a-303ee3dd4d30-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:37.069391 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:37:37.069281    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 23:37:37.069391 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:37:37.069359    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0026773a-150e-4942-8dd1-239fa023a131-proxy-tls podName:0026773a-150e-4942-8dd1-239fa023a131 nodeName:}" failed. No retries permitted until 2026-09-17 23:37:37.569339342 +0000 UTC m=+1590.613715484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0026773a-150e-4942-8dd1-239fa023a131-proxy-tls") pod "isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" (UID: "0026773a-150e-4942-8dd1-239fa023a131") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 23:37:37.069391 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:37:37.069386    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 23:37:37.069867 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:37:37.069436    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f611c5eb-e8f3-491a-962a-303ee3dd4d30-proxy-tls podName:f611c5eb-e8f3-491a-962a-303ee3dd4d30 nodeName:}" failed. No retries permitted until 2026-09-17 23:37:37.569419784 +0000 UTC m=+1590.613795923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f611c5eb-e8f3-491a-962a-303ee3dd4d30-proxy-tls") pod "isvc-canary-create-predictor-65f7bb5954-5ndws" (UID: "f611c5eb-e8f3-491a-962a-303ee3dd4d30") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 23:37:37.069867 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0026773a-150e-4942-8dd1-239fa023a131-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:37.069867 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kserve-provision-location\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:37.069867 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0026773a-150e-4942-8dd1-239fa023a131-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:37.069867 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kserve-provision-location\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:37.070126 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.069908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0026773a-150e-4942-8dd1-239fa023a131-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:37.070126 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.070018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:37.070126 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.070023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0026773a-150e-4942-8dd1-239fa023a131-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:37.070126 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.070070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f611c5eb-e8f3-491a-962a-303ee3dd4d30-cabundle-cert\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:37.080045 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.080012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xx2r\" (UniqueName: \"kubernetes.io/projected/0026773a-150e-4942-8dd1-239fa023a131-kube-api-access-4xx2r\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:37.080182 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.080060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2252\" (UniqueName: \"kubernetes.io/projected/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kube-api-access-h2252\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:37.573719 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.573675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0026773a-150e-4942-8dd1-239fa023a131-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:37.573904 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.573766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f611c5eb-e8f3-491a-962a-303ee3dd4d30-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:37.576285 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.576256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f611c5eb-e8f3-491a-962a-303ee3dd4d30-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-5ndws\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:37.576401 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.576287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0026773a-150e-4942-8dd1-239fa023a131-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:37.840142 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.840063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:37.876235 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:37.876199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:38.117544 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:38.117521    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"]
Sep 17 23:37:38.119792 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:37:38.119757    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0026773a_150e_4942_8dd1_239fa023a131.slice/crio-9480a43040c947c94f8dbaf6571b6fa927deb10815f9a39d06d2dc084727cca8 WatchSource:0}: Error finding container 9480a43040c947c94f8dbaf6571b6fa927deb10815f9a39d06d2dc084727cca8: Status 404 returned error can't find the container with id 9480a43040c947c94f8dbaf6571b6fa927deb10815f9a39d06d2dc084727cca8
Sep 17 23:37:38.154880 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:38.154857    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"]
Sep 17 23:37:38.176761 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:37:38.176726    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf611c5eb_e8f3_491a_962a_303ee3dd4d30.slice/crio-91d4ee8da6cc46f27c857ada4963f24564fb4b80bca90541ac0499ec722c089c WatchSource:0}: Error finding container 91d4ee8da6cc46f27c857ada4963f24564fb4b80bca90541ac0499ec722c089c: Status 404 returned error can't find the container with id 91d4ee8da6cc46f27c857ada4963f24564fb4b80bca90541ac0499ec722c089c
Sep 17 23:37:38.395654 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:38.395542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" event={"ID":"f611c5eb-e8f3-491a-962a-303ee3dd4d30","Type":"ContainerStarted","Data":"fb7deeb124cbe6c4b5b33ca1dc5aa08554f896edabc6039d043b330fef6aec5d"}
Sep 17 23:37:38.395654 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:38.395604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" event={"ID":"f611c5eb-e8f3-491a-962a-303ee3dd4d30","Type":"ContainerStarted","Data":"91d4ee8da6cc46f27c857ada4963f24564fb4b80bca90541ac0499ec722c089c"}
Sep 17 23:37:38.397350 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:38.397318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" event={"ID":"0026773a-150e-4942-8dd1-239fa023a131","Type":"ContainerStarted","Data":"24c62084155fc189f1d28bfdb166b44ae25dd7df8f37dd6f7682b23838fdd84f"}
Sep 17 23:37:38.397482 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:38.397373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" event={"ID":"0026773a-150e-4942-8dd1-239fa023a131","Type":"ContainerStarted","Data":"9480a43040c947c94f8dbaf6571b6fa927deb10815f9a39d06d2dc084727cca8"}
Sep 17 23:37:42.413262 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:42.413228    2580 generic.go:356] "Generic (PLEG): container finished" podID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerID="fb7deeb124cbe6c4b5b33ca1dc5aa08554f896edabc6039d043b330fef6aec5d" exitCode=0
Sep 17 23:37:42.413693 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:42.413306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" event={"ID":"f611c5eb-e8f3-491a-962a-303ee3dd4d30","Type":"ContainerDied","Data":"fb7deeb124cbe6c4b5b33ca1dc5aa08554f896edabc6039d043b330fef6aec5d"}
Sep 17 23:37:42.414607 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:42.414587    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:37:42.414843 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:42.414819    2580 generic.go:356] "Generic (PLEG): container finished" podID="0026773a-150e-4942-8dd1-239fa023a131" containerID="24c62084155fc189f1d28bfdb166b44ae25dd7df8f37dd6f7682b23838fdd84f" exitCode=0
Sep 17 23:37:42.414921 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:42.414888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" event={"ID":"0026773a-150e-4942-8dd1-239fa023a131","Type":"ContainerDied","Data":"24c62084155fc189f1d28bfdb166b44ae25dd7df8f37dd6f7682b23838fdd84f"}
Sep 17 23:37:59.512699 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:59.512658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" event={"ID":"f611c5eb-e8f3-491a-962a-303ee3dd4d30","Type":"ContainerStarted","Data":"f74a669d5dae045ca19d13ef75ca47736e6275eec934e0c9a87ec84a29b4ad4e"}
Sep 17 23:37:59.513186 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:59.512734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" event={"ID":"f611c5eb-e8f3-491a-962a-303ee3dd4d30","Type":"ContainerStarted","Data":"db7e7f927160f23684fadb2a05fb029c970304fc895ceb3c3efc5419bda61b69"}
Sep 17 23:37:59.513186 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:59.512934    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:37:59.514815 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:59.514797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" event={"ID":"0026773a-150e-4942-8dd1-239fa023a131","Type":"ContainerStarted","Data":"1bd56a3c60ac4e90c672bc951f3f00e16079a60ff605a14363452442d779b5d0"}
Sep 17 23:37:59.514868 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:59.514842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" event={"ID":"0026773a-150e-4942-8dd1-239fa023a131","Type":"ContainerStarted","Data":"1e530cbf6be91dd02beda49c4744900ce0a0548513202db090925702ac32ab74"}
Sep 17 23:37:59.515077 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:59.515061    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:59.515112 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:59.515103    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:37:59.516131 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:59.516108    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 23:37:59.532511 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:59.532466    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podStartSLOduration=3.3756442890000002 podStartE2EDuration="23.532451626s" totalImagesPullingTime="16.16258778s" totalInitContainerRuntime="3.994219557s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:59.530596131 +0000 UTC m=+1612.574972292" watchObservedRunningTime="2026-09-17 23:37:59.532451626 +0000 UTC m=+1612.576827786"
Sep 17 23:37:59.549482 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:37:59.549434    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podStartSLOduration=3.67211809 podStartE2EDuration="23.549420774s" totalImagesPullingTime="16.181244789s" totalInitContainerRuntime="3.696057895s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:59.548030432 +0000 UTC m=+1612.592406591" watchObservedRunningTime="2026-09-17 23:37:59.549420774 +0000 UTC m=+1612.593796934"
Sep 17 23:38:00.519091 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:00.519045    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 23:38:00.519517 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:00.519263    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:38:00.520355 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:00.520335    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:38:01.522306 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:01.522258    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:38:05.524389 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:05.524362    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:38:05.524919 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:05.524893    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 23:38:06.526633 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:06.526583    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:38:06.527113 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:06.527088    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:38:15.525510 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:15.525471    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 23:38:16.527592 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:16.527540    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:38:25.525183 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:25.525145    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 23:38:26.528056 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:26.528017    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:38:35.525117 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:35.525080    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 23:38:36.527532 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:36.527489    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:38:45.525403 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:45.525360    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 23:38:46.528051 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:46.528014    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:38:55.525250 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:55.525207    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 23:38:56.527807 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:38:56.527768    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:39:05.524791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:05.524755    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 23:39:06.527322 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:06.527280    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:39:08.288895 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.288859    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"]
Sep 17 23:39:08.292144 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.292124    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.294729 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.294708    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 17 23:39:08.294729 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.294724    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 23:39:08.305311 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.305286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"]
Sep 17 23:39:08.373985 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.373943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a699a566-7f9c-4105-9b99-2df7f6a93871-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.374151 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.373999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a699a566-7f9c-4105-9b99-2df7f6a93871-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.374151 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.374086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a699a566-7f9c-4105-9b99-2df7f6a93871-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.374151 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.374131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g59j\" (UniqueName: \"kubernetes.io/projected/a699a566-7f9c-4105-9b99-2df7f6a93871-kube-api-access-9g59j\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.374255 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.374198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a699a566-7f9c-4105-9b99-2df7f6a93871-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.474978 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.474935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a699a566-7f9c-4105-9b99-2df7f6a93871-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.475154 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.474986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g59j\" (UniqueName: \"kubernetes.io/projected/a699a566-7f9c-4105-9b99-2df7f6a93871-kube-api-access-9g59j\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.475154 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.475033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a699a566-7f9c-4105-9b99-2df7f6a93871-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.475154 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.475086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a699a566-7f9c-4105-9b99-2df7f6a93871-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.475154 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.475112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a699a566-7f9c-4105-9b99-2df7f6a93871-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.475777 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.475750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a699a566-7f9c-4105-9b99-2df7f6a93871-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.475894 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.475792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a699a566-7f9c-4105-9b99-2df7f6a93871-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.475894 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.475813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a699a566-7f9c-4105-9b99-2df7f6a93871-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.477660 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.477645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a699a566-7f9c-4105-9b99-2df7f6a93871-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.483866 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.483843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g59j\" (UniqueName: \"kubernetes.io/projected/a699a566-7f9c-4105-9b99-2df7f6a93871-kube-api-access-9g59j\") pod \"isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.602031 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.601937    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:08.875532 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:08.875508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"]
Sep 17 23:39:08.878124 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:39:08.878100    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda699a566_7f9c_4105_9b99_2df7f6a93871.slice/crio-3e26dee46945136acab8f87a90c6938a140954d3445c717ab6a550728bc583c5 WatchSource:0}: Error finding container 3e26dee46945136acab8f87a90c6938a140954d3445c717ab6a550728bc583c5: Status 404 returned error can't find the container with id 3e26dee46945136acab8f87a90c6938a140954d3445c717ab6a550728bc583c5
Sep 17 23:39:09.788862 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:09.788825    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" event={"ID":"a699a566-7f9c-4105-9b99-2df7f6a93871","Type":"ContainerStarted","Data":"1b114e2a8208be4a952802fb94cf6d8f7e5bc061406eae37706d1ce66df6bfb5"}
Sep 17 23:39:09.789288 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:09.788886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" event={"ID":"a699a566-7f9c-4105-9b99-2df7f6a93871","Type":"ContainerStarted","Data":"3e26dee46945136acab8f87a90c6938a140954d3445c717ab6a550728bc583c5"}
Sep 17 23:39:12.800809 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:12.800770    2580 generic.go:356] "Generic (PLEG): container finished" podID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerID="1b114e2a8208be4a952802fb94cf6d8f7e5bc061406eae37706d1ce66df6bfb5" exitCode=0
Sep 17 23:39:12.801204 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:12.800843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" event={"ID":"a699a566-7f9c-4105-9b99-2df7f6a93871","Type":"ContainerDied","Data":"1b114e2a8208be4a952802fb94cf6d8f7e5bc061406eae37706d1ce66df6bfb5"}
Sep 17 23:39:13.806991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:13.806956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" event={"ID":"a699a566-7f9c-4105-9b99-2df7f6a93871","Type":"ContainerStarted","Data":"9f5ceb18f8e8b4f125f10b1832176b5866a5e63b3b13c99438453cb4c6a8c611"}
Sep 17 23:39:13.807397 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:13.807025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" event={"ID":"a699a566-7f9c-4105-9b99-2df7f6a93871","Type":"ContainerStarted","Data":"b58ad69c08c1ed61254fedb59b7be413ac0c98e244c86eb072544b0591d9e916"}
Sep 17 23:39:13.807397 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:13.807243    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:13.826648 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:13.826577    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podStartSLOduration=2.078198129 podStartE2EDuration="5.826558684s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.748360555s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:39:13.824773976 +0000 UTC m=+1686.869150136" watchObservedRunningTime="2026-09-17 23:39:13.826558684 +0000 UTC m=+1686.870934843"
Sep 17 23:39:14.811549 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:14.811513    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:14.812819 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:14.812776    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:39:15.525670 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:15.525644    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:39:15.645692 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:15.645653    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"]
Sep 17 23:39:15.645939 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:15.645915    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container" containerID="cri-o://db7e7f927160f23684fadb2a05fb029c970304fc895ceb3c3efc5419bda61b69" gracePeriod=30
Sep 17 23:39:15.646022 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:15.645941    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kube-rbac-proxy" containerID="cri-o://f74a669d5dae045ca19d13ef75ca47736e6275eec934e0c9a87ec84a29b4ad4e" gracePeriod=30
Sep 17 23:39:15.646806 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:15.646779    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:39:15.817406 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:15.817369    2580 generic.go:356] "Generic (PLEG): container finished" podID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerID="f74a669d5dae045ca19d13ef75ca47736e6275eec934e0c9a87ec84a29b4ad4e" exitCode=2
Sep 17 23:39:15.817821 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:15.817445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" event={"ID":"f611c5eb-e8f3-491a-962a-303ee3dd4d30","Type":"ContainerDied","Data":"f74a669d5dae045ca19d13ef75ca47736e6275eec934e0c9a87ec84a29b4ad4e"}
Sep 17 23:39:15.817821 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:15.817729    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:39:16.522936 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:16.522893    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 23:39:17.009001 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:17.008964    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"]
Sep 17 23:39:17.009428 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:17.009339    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container" containerID="cri-o://1e530cbf6be91dd02beda49c4744900ce0a0548513202db090925702ac32ab74" gracePeriod=30
Sep 17 23:39:17.009734 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:17.009677    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kube-rbac-proxy" containerID="cri-o://1bd56a3c60ac4e90c672bc951f3f00e16079a60ff605a14363452442d779b5d0" gracePeriod=30
Sep 17 23:39:17.826110 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:17.826079    2580 generic.go:356] "Generic (PLEG): container finished" podID="0026773a-150e-4942-8dd1-239fa023a131" containerID="1bd56a3c60ac4e90c672bc951f3f00e16079a60ff605a14363452442d779b5d0" exitCode=2
Sep 17 23:39:17.826362 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:17.826148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" event={"ID":"0026773a-150e-4942-8dd1-239fa023a131","Type":"ContainerDied","Data":"1bd56a3c60ac4e90c672bc951f3f00e16079a60ff605a14363452442d779b5d0"}
Sep 17 23:39:20.519728 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.519687    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 17 23:39:20.822830 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.822809    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:39:20.823482 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.823448    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:39:20.840598 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.840571    2580 generic.go:356] "Generic (PLEG): container finished" podID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerID="db7e7f927160f23684fadb2a05fb029c970304fc895ceb3c3efc5419bda61b69" exitCode=0
Sep 17 23:39:20.840823 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.840652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" event={"ID":"f611c5eb-e8f3-491a-962a-303ee3dd4d30","Type":"ContainerDied","Data":"db7e7f927160f23684fadb2a05fb029c970304fc895ceb3c3efc5419bda61b69"}
Sep 17 23:39:20.888715 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.888691    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:39:20.976298 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.976270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f611c5eb-e8f3-491a-962a-303ee3dd4d30-cabundle-cert\") pod \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") "
Sep 17 23:39:20.976298 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.976309    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kube-rbac-proxy-sar-config\") pod \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") "
Sep 17 23:39:20.976491 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.976342    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2252\" (UniqueName: \"kubernetes.io/projected/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kube-api-access-h2252\") pod \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") "
Sep 17 23:39:20.976491 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.976370    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kserve-provision-location\") pod \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") "
Sep 17 23:39:20.976491 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.976390    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f611c5eb-e8f3-491a-962a-303ee3dd4d30-proxy-tls\") pod \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\" (UID: \"f611c5eb-e8f3-491a-962a-303ee3dd4d30\") "
Sep 17 23:39:20.976708 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.976683    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f611c5eb-e8f3-491a-962a-303ee3dd4d30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f611c5eb-e8f3-491a-962a-303ee3dd4d30" (UID: "f611c5eb-e8f3-491a-962a-303ee3dd4d30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:20.976746 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.976700    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f611c5eb-e8f3-491a-962a-303ee3dd4d30" (UID: "f611c5eb-e8f3-491a-962a-303ee3dd4d30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:20.976746 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.976705    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f611c5eb-e8f3-491a-962a-303ee3dd4d30" (UID: "f611c5eb-e8f3-491a-962a-303ee3dd4d30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:20.978551 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.978529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kube-api-access-h2252" (OuterVolumeSpecName: "kube-api-access-h2252") pod "f611c5eb-e8f3-491a-962a-303ee3dd4d30" (UID: "f611c5eb-e8f3-491a-962a-303ee3dd4d30"). InnerVolumeSpecName "kube-api-access-h2252". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:20.978687 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:20.978670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f611c5eb-e8f3-491a-962a-303ee3dd4d30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f611c5eb-e8f3-491a-962a-303ee3dd4d30" (UID: "f611c5eb-e8f3-491a-962a-303ee3dd4d30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:39:21.077897 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:21.077812    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f611c5eb-e8f3-491a-962a-303ee3dd4d30-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:39:21.077897 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:21.077841    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:39:21.077897 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:21.077851    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2252\" (UniqueName: \"kubernetes.io/projected/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kube-api-access-h2252\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:39:21.077897 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:21.077860    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f611c5eb-e8f3-491a-962a-303ee3dd4d30-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:39:21.077897 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:21.077869    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f611c5eb-e8f3-491a-962a-303ee3dd4d30-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:39:21.846316 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:21.846289    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"
Sep 17 23:39:21.846803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:21.846282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws" event={"ID":"f611c5eb-e8f3-491a-962a-303ee3dd4d30","Type":"ContainerDied","Data":"91d4ee8da6cc46f27c857ada4963f24564fb4b80bca90541ac0499ec722c089c"}
Sep 17 23:39:21.846803 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:21.846426    2580 scope.go:117] "RemoveContainer" containerID="f74a669d5dae045ca19d13ef75ca47736e6275eec934e0c9a87ec84a29b4ad4e"
Sep 17 23:39:21.854370 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:21.854352    2580 scope.go:117] "RemoveContainer" containerID="db7e7f927160f23684fadb2a05fb029c970304fc895ceb3c3efc5419bda61b69"
Sep 17 23:39:21.861532 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:21.861501    2580 scope.go:117] "RemoveContainer" containerID="fb7deeb124cbe6c4b5b33ca1dc5aa08554f896edabc6039d043b330fef6aec5d"
Sep 17 23:39:21.868002 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:21.867969    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"]
Sep 17 23:39:21.872143 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:21.872115    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5ndws"]
Sep 17 23:39:22.217989 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.217917    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"]
Sep 17 23:39:22.218220 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.218207    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container"
Sep 17 23:39:22.218261 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.218222    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container"
Sep 17 23:39:22.218261 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.218238    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kube-rbac-proxy"
Sep 17 23:39:22.218261 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.218246    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kube-rbac-proxy"
Sep 17 23:39:22.218353 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.218261    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="storage-initializer"
Sep 17 23:39:22.218353 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.218270    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="storage-initializer"
Sep 17 23:39:22.218353 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.218316    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kserve-container"
Sep 17 23:39:22.218353 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.218325    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" containerName="kube-rbac-proxy"
Sep 17 23:39:22.222697 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.222680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.225256 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.225233    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 23:39:22.225388 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.225271    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 23:39:22.231885 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.231863    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"]
Sep 17 23:39:22.287364 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.287328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kserve-provision-location\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.287541 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.287373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72ttf\" (UniqueName: \"kubernetes.io/projected/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kube-api-access-72ttf\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.287541 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.287396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-cabundle-cert\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.287541 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.287492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.287676 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.287548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.388904 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.388869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.388904 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.388909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.389129 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.388957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kserve-provision-location\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.389129 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.388973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72ttf\" (UniqueName: \"kubernetes.io/projected/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kube-api-access-72ttf\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.389129 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.388991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-cabundle-cert\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.389129 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:39:22.389117    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 23:39:22.389353 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:39:22.389205    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-proxy-tls podName:11dd5cb6-7a7f-42c2-9829-9abb5b70f585 nodeName:}" failed. No retries permitted until 2026-09-17 23:39:22.889186238 +0000 UTC m=+1695.933562376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-proxy-tls") pod "isvc-canary-promote-predictor-fc78f5cb6-rzrjn" (UID: "11dd5cb6-7a7f-42c2-9829-9abb5b70f585") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 23:39:22.389490 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.389432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kserve-provision-location\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.389684 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.389669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-cabundle-cert\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.389732 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.389708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.397598 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.397571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72ttf\" (UniqueName: \"kubernetes.io/projected/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kube-api-access-72ttf\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.893174 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.893142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:22.895683 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:22.895666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-rzrjn\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:23.132530 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:23.132492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:23.407967 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:23.407934    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"]
Sep 17 23:39:23.411173 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:39:23.411143    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dd5cb6_7a7f_42c2_9829_9abb5b70f585.slice/crio-b19df30aa2a08aa211cc71c59962e27cb40fda33df1c5f0d7c158980babc7498 WatchSource:0}: Error finding container b19df30aa2a08aa211cc71c59962e27cb40fda33df1c5f0d7c158980babc7498: Status 404 returned error can't find the container with id b19df30aa2a08aa211cc71c59962e27cb40fda33df1c5f0d7c158980babc7498
Sep 17 23:39:23.502035 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:23.502006    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f611c5eb-e8f3-491a-962a-303ee3dd4d30" path="/var/lib/kubelet/pods/f611c5eb-e8f3-491a-962a-303ee3dd4d30/volumes"
Sep 17 23:39:23.856925 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:23.856891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" event={"ID":"11dd5cb6-7a7f-42c2-9829-9abb5b70f585","Type":"ContainerStarted","Data":"e5087b4b4014cc3bea841793d92516c6b707589c161bf01ba2bf7702a9df6cc0"}
Sep 17 23:39:23.857100 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:23.856963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" event={"ID":"11dd5cb6-7a7f-42c2-9829-9abb5b70f585","Type":"ContainerStarted","Data":"b19df30aa2a08aa211cc71c59962e27cb40fda33df1c5f0d7c158980babc7498"}
Sep 17 23:39:23.859213 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:23.859183    2580 generic.go:356] "Generic (PLEG): container finished" podID="0026773a-150e-4942-8dd1-239fa023a131" containerID="1e530cbf6be91dd02beda49c4744900ce0a0548513202db090925702ac32ab74" exitCode=0
Sep 17 23:39:23.859338 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:23.859221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" event={"ID":"0026773a-150e-4942-8dd1-239fa023a131","Type":"ContainerDied","Data":"1e530cbf6be91dd02beda49c4744900ce0a0548513202db090925702ac32ab74"}
Sep 17 23:39:23.959946 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:23.959924    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:39:24.103860 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.103828    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0026773a-150e-4942-8dd1-239fa023a131-cabundle-cert\") pod \"0026773a-150e-4942-8dd1-239fa023a131\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") "
Sep 17 23:39:24.104071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.103874    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xx2r\" (UniqueName: \"kubernetes.io/projected/0026773a-150e-4942-8dd1-239fa023a131-kube-api-access-4xx2r\") pod \"0026773a-150e-4942-8dd1-239fa023a131\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") "
Sep 17 23:39:24.104071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.103900    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0026773a-150e-4942-8dd1-239fa023a131-kube-rbac-proxy-sar-config\") pod \"0026773a-150e-4942-8dd1-239fa023a131\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") "
Sep 17 23:39:24.104071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.103934    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0026773a-150e-4942-8dd1-239fa023a131-proxy-tls\") pod \"0026773a-150e-4942-8dd1-239fa023a131\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") "
Sep 17 23:39:24.104071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.104005    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0026773a-150e-4942-8dd1-239fa023a131-kserve-provision-location\") pod \"0026773a-150e-4942-8dd1-239fa023a131\" (UID: \"0026773a-150e-4942-8dd1-239fa023a131\") "
Sep 17 23:39:24.104318 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.104256    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0026773a-150e-4942-8dd1-239fa023a131-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0026773a-150e-4942-8dd1-239fa023a131" (UID: "0026773a-150e-4942-8dd1-239fa023a131"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:24.104391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.104364    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0026773a-150e-4942-8dd1-239fa023a131-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0026773a-150e-4942-8dd1-239fa023a131" (UID: "0026773a-150e-4942-8dd1-239fa023a131"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:24.104391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.104275    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0026773a-150e-4942-8dd1-239fa023a131-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0026773a-150e-4942-8dd1-239fa023a131" (UID: "0026773a-150e-4942-8dd1-239fa023a131"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:24.106133 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.106112    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0026773a-150e-4942-8dd1-239fa023a131-kube-api-access-4xx2r" (OuterVolumeSpecName: "kube-api-access-4xx2r") pod "0026773a-150e-4942-8dd1-239fa023a131" (UID: "0026773a-150e-4942-8dd1-239fa023a131"). InnerVolumeSpecName "kube-api-access-4xx2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:24.106242 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.106223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0026773a-150e-4942-8dd1-239fa023a131-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0026773a-150e-4942-8dd1-239fa023a131" (UID: "0026773a-150e-4942-8dd1-239fa023a131"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:39:24.204782 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.204747    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0026773a-150e-4942-8dd1-239fa023a131-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:39:24.204782 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.204773    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0026773a-150e-4942-8dd1-239fa023a131-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:39:24.204782 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.204782    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xx2r\" (UniqueName: \"kubernetes.io/projected/0026773a-150e-4942-8dd1-239fa023a131-kube-api-access-4xx2r\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:39:24.204782 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.204790    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0026773a-150e-4942-8dd1-239fa023a131-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:39:24.205025 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.204802    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0026773a-150e-4942-8dd1-239fa023a131-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:39:24.865203 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.865164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr" event={"ID":"0026773a-150e-4942-8dd1-239fa023a131","Type":"ContainerDied","Data":"9480a43040c947c94f8dbaf6571b6fa927deb10815f9a39d06d2dc084727cca8"}
Sep 17 23:39:24.865356 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.865212    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"
Sep 17 23:39:24.865356 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.865241    2580 scope.go:117] "RemoveContainer" containerID="1bd56a3c60ac4e90c672bc951f3f00e16079a60ff605a14363452442d779b5d0"
Sep 17 23:39:24.876866 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.876841    2580 scope.go:117] "RemoveContainer" containerID="1e530cbf6be91dd02beda49c4744900ce0a0548513202db090925702ac32ab74"
Sep 17 23:39:24.884346 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.884326    2580 scope.go:117] "RemoveContainer" containerID="24c62084155fc189f1d28bfdb166b44ae25dd7df8f37dd6f7682b23838fdd84f"
Sep 17 23:39:24.892380 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.892362    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"]
Sep 17 23:39:24.896731 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:24.896709    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-gcrhr"]
Sep 17 23:39:25.500254 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:25.500221    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0026773a-150e-4942-8dd1-239fa023a131" path="/var/lib/kubelet/pods/0026773a-150e-4942-8dd1-239fa023a131/volumes"
Sep 17 23:39:27.879885 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:27.879855    2580 generic.go:356] "Generic (PLEG): container finished" podID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerID="e5087b4b4014cc3bea841793d92516c6b707589c161bf01ba2bf7702a9df6cc0" exitCode=0
Sep 17 23:39:27.880335 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:27.879920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" event={"ID":"11dd5cb6-7a7f-42c2-9829-9abb5b70f585","Type":"ContainerDied","Data":"e5087b4b4014cc3bea841793d92516c6b707589c161bf01ba2bf7702a9df6cc0"}
Sep 17 23:39:28.885385 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:28.885313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" event={"ID":"11dd5cb6-7a7f-42c2-9829-9abb5b70f585","Type":"ContainerStarted","Data":"7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b"}
Sep 17 23:39:28.885385 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:28.885370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" event={"ID":"11dd5cb6-7a7f-42c2-9829-9abb5b70f585","Type":"ContainerStarted","Data":"0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75"}
Sep 17 23:39:28.885815 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:28.885666    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:28.885851 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:28.885813    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:28.886946 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:28.886923    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:39:28.903680 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:28.903631    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podStartSLOduration=3.1250574110000002 podStartE2EDuration="6.9035953s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.778537889s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:39:28.90326183 +0000 UTC m=+1701.947637990" watchObservedRunningTime="2026-09-17 23:39:28.9035953 +0000 UTC m=+1701.947971460"
Sep 17 23:39:29.888549 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:29.888513    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:39:30.823505 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:30.823467    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:39:34.893307 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:34.893280    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:39:34.893949 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:34.893919    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:39:40.823738 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:40.823693    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:39:44.894765 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:44.894726    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:39:50.824249 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:50.824212    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:39:54.894295 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:39:54.894250    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:40:00.823995 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:00.823952    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:40:04.893854 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:04.893813    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:40:10.824147 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:10.824105    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:40:14.894214 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:14.894176    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:40:20.823565 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:20.823526    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:40:24.894050 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:24.894009    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:40:30.823818 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:30.823786    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:40:34.893962 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:34.893925    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:40:38.463012 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:38.462971    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"]
Sep 17 23:40:38.463391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:38.463286    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container" containerID="cri-o://b58ad69c08c1ed61254fedb59b7be413ac0c98e244c86eb072544b0591d9e916" gracePeriod=30
Sep 17 23:40:38.463391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:38.463295    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kube-rbac-proxy" containerID="cri-o://9f5ceb18f8e8b4f125f10b1832176b5866a5e63b3b13c99438453cb4c6a8c611" gracePeriod=30
Sep 17 23:40:39.152877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:39.152846    2580 generic.go:356] "Generic (PLEG): container finished" podID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerID="9f5ceb18f8e8b4f125f10b1832176b5866a5e63b3b13c99438453cb4c6a8c611" exitCode=2
Sep 17 23:40:39.153047 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:39.152927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" event={"ID":"a699a566-7f9c-4105-9b99-2df7f6a93871","Type":"ContainerDied","Data":"9f5ceb18f8e8b4f125f10b1832176b5866a5e63b3b13c99438453cb4c6a8c611"}
Sep 17 23:40:40.818066 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:40.818022    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 23:40:40.823374 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:40.823350    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:40:43.168210 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.168175    2580 generic.go:356] "Generic (PLEG): container finished" podID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerID="b58ad69c08c1ed61254fedb59b7be413ac0c98e244c86eb072544b0591d9e916" exitCode=0
Sep 17 23:40:43.168210 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.168202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" event={"ID":"a699a566-7f9c-4105-9b99-2df7f6a93871","Type":"ContainerDied","Data":"b58ad69c08c1ed61254fedb59b7be413ac0c98e244c86eb072544b0591d9e916"}
Sep 17 23:40:43.199496 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.199469    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:40:43.302895 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.302869    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a699a566-7f9c-4105-9b99-2df7f6a93871-cabundle-cert\") pod \"a699a566-7f9c-4105-9b99-2df7f6a93871\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") "
Sep 17 23:40:43.303030 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.302908    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g59j\" (UniqueName: \"kubernetes.io/projected/a699a566-7f9c-4105-9b99-2df7f6a93871-kube-api-access-9g59j\") pod \"a699a566-7f9c-4105-9b99-2df7f6a93871\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") "
Sep 17 23:40:43.303030 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.302934    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a699a566-7f9c-4105-9b99-2df7f6a93871-kserve-provision-location\") pod \"a699a566-7f9c-4105-9b99-2df7f6a93871\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") "
Sep 17 23:40:43.303030 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.302965    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a699a566-7f9c-4105-9b99-2df7f6a93871-proxy-tls\") pod \"a699a566-7f9c-4105-9b99-2df7f6a93871\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") "
Sep 17 23:40:43.303030 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.303003    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a699a566-7f9c-4105-9b99-2df7f6a93871-kube-rbac-proxy-sar-config\") pod \"a699a566-7f9c-4105-9b99-2df7f6a93871\" (UID: \"a699a566-7f9c-4105-9b99-2df7f6a93871\") "
Sep 17 23:40:43.303271 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.303240    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a699a566-7f9c-4105-9b99-2df7f6a93871-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a699a566-7f9c-4105-9b99-2df7f6a93871" (UID: "a699a566-7f9c-4105-9b99-2df7f6a93871"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:40:43.303388 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.303264    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a699a566-7f9c-4105-9b99-2df7f6a93871-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a699a566-7f9c-4105-9b99-2df7f6a93871" (UID: "a699a566-7f9c-4105-9b99-2df7f6a93871"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:40:43.303452 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.303396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a699a566-7f9c-4105-9b99-2df7f6a93871-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a699a566-7f9c-4105-9b99-2df7f6a93871" (UID: "a699a566-7f9c-4105-9b99-2df7f6a93871"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:40:43.305142 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.305122    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a699a566-7f9c-4105-9b99-2df7f6a93871-kube-api-access-9g59j" (OuterVolumeSpecName: "kube-api-access-9g59j") pod "a699a566-7f9c-4105-9b99-2df7f6a93871" (UID: "a699a566-7f9c-4105-9b99-2df7f6a93871"). InnerVolumeSpecName "kube-api-access-9g59j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:40:43.305216 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.305167    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a699a566-7f9c-4105-9b99-2df7f6a93871-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a699a566-7f9c-4105-9b99-2df7f6a93871" (UID: "a699a566-7f9c-4105-9b99-2df7f6a93871"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:40:43.404436 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.404400    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g59j\" (UniqueName: \"kubernetes.io/projected/a699a566-7f9c-4105-9b99-2df7f6a93871-kube-api-access-9g59j\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:40:43.404436 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.404428    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a699a566-7f9c-4105-9b99-2df7f6a93871-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:40:43.404436 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.404438    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a699a566-7f9c-4105-9b99-2df7f6a93871-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:40:43.404669 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.404449    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a699a566-7f9c-4105-9b99-2df7f6a93871-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:40:43.404669 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.404457    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a699a566-7f9c-4105-9b99-2df7f6a93871-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:40:43.650677 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650591    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"]
Sep 17 23:40:43.650871 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650859    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container"
Sep 17 23:40:43.650913 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650872    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container"
Sep 17 23:40:43.650913 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650881    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kube-rbac-proxy"
Sep 17 23:40:43.650913 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650887    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kube-rbac-proxy"
Sep 17 23:40:43.650913 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650893    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kube-rbac-proxy"
Sep 17 23:40:43.650913 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650898    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kube-rbac-proxy"
Sep 17 23:40:43.650913 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650907    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="storage-initializer"
Sep 17 23:40:43.650913 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650912    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="storage-initializer"
Sep 17 23:40:43.651114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650923    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container"
Sep 17 23:40:43.651114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650928    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container"
Sep 17 23:40:43.651114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650938    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="storage-initializer"
Sep 17 23:40:43.651114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650943    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="storage-initializer"
Sep 17 23:40:43.651114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650984    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kube-rbac-proxy"
Sep 17 23:40:43.651114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650993    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kserve-container"
Sep 17 23:40:43.651114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.650998    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" containerName="kube-rbac-proxy"
Sep 17 23:40:43.651114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.651004    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0026773a-150e-4942-8dd1-239fa023a131" containerName="kserve-container"
Sep 17 23:40:43.654211 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.654193    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.656410 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.656391    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 17 23:40:43.656592 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.656574    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 17 23:40:43.664936 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.664915    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"]
Sep 17 23:40:43.807026 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.806988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.807209 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.807043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86daa65b-d287-4f57-8e60-b63c9b8f29aa-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.807209 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.807066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.807209 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.807101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86daa65b-d287-4f57-8e60-b63c9b8f29aa-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.807209 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.807196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7ttd\" (UniqueName: \"kubernetes.io/projected/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kube-api-access-c7ttd\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.908513 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.908441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ttd\" (UniqueName: \"kubernetes.io/projected/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kube-api-access-c7ttd\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.908513 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.908481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.908690 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.908633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86daa65b-d287-4f57-8e60-b63c9b8f29aa-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.908690 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.908680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.908758 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.908706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86daa65b-d287-4f57-8e60-b63c9b8f29aa-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.908821 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.908805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.909420 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.909353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.909420 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.909360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86daa65b-d287-4f57-8e60-b63c9b8f29aa-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.911209 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.911189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86daa65b-d287-4f57-8e60-b63c9b8f29aa-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.917139 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.917122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ttd\" (UniqueName: \"kubernetes.io/projected/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kube-api-access-c7ttd\") pod \"isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:43.964588 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:43.964564    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:44.175728 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.175638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt" event={"ID":"a699a566-7f9c-4105-9b99-2df7f6a93871","Type":"ContainerDied","Data":"3e26dee46945136acab8f87a90c6938a140954d3445c717ab6a550728bc583c5"}
Sep 17 23:40:44.175728 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.175679    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"
Sep 17 23:40:44.175728 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.175723    2580 scope.go:117] "RemoveContainer" containerID="9f5ceb18f8e8b4f125f10b1832176b5866a5e63b3b13c99438453cb4c6a8c611"
Sep 17 23:40:44.183461 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.183440    2580 scope.go:117] "RemoveContainer" containerID="b58ad69c08c1ed61254fedb59b7be413ac0c98e244c86eb072544b0591d9e916"
Sep 17 23:40:44.190606 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.190588    2580 scope.go:117] "RemoveContainer" containerID="1b114e2a8208be4a952802fb94cf6d8f7e5bc061406eae37706d1ce66df6bfb5"
Sep 17 23:40:44.196034 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.196006    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"]
Sep 17 23:40:44.200657 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.200636    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cf4cf69ff-pq4bt"]
Sep 17 23:40:44.240725 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.240703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"]
Sep 17 23:40:44.243109 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:40:44.243079    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86daa65b_d287_4f57_8e60_b63c9b8f29aa.slice/crio-89ade21a7d5f70adff3897f89aedef5e5c8b3658ec3533dfa3dd5bbfa2e95772 WatchSource:0}: Error finding container 89ade21a7d5f70adff3897f89aedef5e5c8b3658ec3533dfa3dd5bbfa2e95772: Status 404 returned error can't find the container with id 89ade21a7d5f70adff3897f89aedef5e5c8b3658ec3533dfa3dd5bbfa2e95772
Sep 17 23:40:44.693411 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.693383    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"]
Sep 17 23:40:44.693702 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.693675    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container" containerID="cri-o://0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75" gracePeriod=30
Sep 17 23:40:44.693813 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.693704    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kube-rbac-proxy" containerID="cri-o://7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b" gracePeriod=30
Sep 17 23:40:44.694537 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.694520    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:40:44.888897 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:44.888860    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 23:40:45.182468 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:45.182439    2580 generic.go:356] "Generic (PLEG): container finished" podID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerID="7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b" exitCode=2
Sep 17 23:40:45.182976 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:45.182519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" event={"ID":"11dd5cb6-7a7f-42c2-9829-9abb5b70f585","Type":"ContainerDied","Data":"7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b"}
Sep 17 23:40:45.183976 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:45.183953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" event={"ID":"86daa65b-d287-4f57-8e60-b63c9b8f29aa","Type":"ContainerStarted","Data":"cd455e1749799fe4bc9748201080d1267b2c67532e579e76cd952799c8438b1c"}
Sep 17 23:40:45.184086 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:45.184015    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" event={"ID":"86daa65b-d287-4f57-8e60-b63c9b8f29aa","Type":"ContainerStarted","Data":"89ade21a7d5f70adff3897f89aedef5e5c8b3658ec3533dfa3dd5bbfa2e95772"}
Sep 17 23:40:45.500878 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:45.500788    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a699a566-7f9c-4105-9b99-2df7f6a93871" path="/var/lib/kubelet/pods/a699a566-7f9c-4105-9b99-2df7f6a93871/volumes"
Sep 17 23:40:49.199132 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.199101    2580 generic.go:356] "Generic (PLEG): container finished" podID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerID="cd455e1749799fe4bc9748201080d1267b2c67532e579e76cd952799c8438b1c" exitCode=0
Sep 17 23:40:49.199585 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.199183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" event={"ID":"86daa65b-d287-4f57-8e60-b63c9b8f29aa","Type":"ContainerDied","Data":"cd455e1749799fe4bc9748201080d1267b2c67532e579e76cd952799c8438b1c"}
Sep 17 23:40:49.435237 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.435213    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:40:49.553442 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.553416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kube-rbac-proxy-sar-config\") pod \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") "
Sep 17 23:40:49.553637 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.553456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72ttf\" (UniqueName: \"kubernetes.io/projected/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kube-api-access-72ttf\") pod \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") "
Sep 17 23:40:49.553637 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.553482    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kserve-provision-location\") pod \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") "
Sep 17 23:40:49.553637 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.553497    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-cabundle-cert\") pod \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") "
Sep 17 23:40:49.553637 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.553514    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-proxy-tls\") pod \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\" (UID: \"11dd5cb6-7a7f-42c2-9829-9abb5b70f585\") "
Sep 17 23:40:49.553853 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.553827    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "11dd5cb6-7a7f-42c2-9829-9abb5b70f585" (UID: "11dd5cb6-7a7f-42c2-9829-9abb5b70f585"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:40:49.553853 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.553840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "11dd5cb6-7a7f-42c2-9829-9abb5b70f585" (UID: "11dd5cb6-7a7f-42c2-9829-9abb5b70f585"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:40:49.553933 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.553868    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "11dd5cb6-7a7f-42c2-9829-9abb5b70f585" (UID: "11dd5cb6-7a7f-42c2-9829-9abb5b70f585"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:40:49.555779 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.555756    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11dd5cb6-7a7f-42c2-9829-9abb5b70f585" (UID: "11dd5cb6-7a7f-42c2-9829-9abb5b70f585"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:40:49.555882 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.555781    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kube-api-access-72ttf" (OuterVolumeSpecName: "kube-api-access-72ttf") pod "11dd5cb6-7a7f-42c2-9829-9abb5b70f585" (UID: "11dd5cb6-7a7f-42c2-9829-9abb5b70f585"). InnerVolumeSpecName "kube-api-access-72ttf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:40:49.654287 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.654253    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:40:49.654287 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.654281    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:40:49.654287 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.654290    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72ttf\" (UniqueName: \"kubernetes.io/projected/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kube-api-access-72ttf\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:40:49.654509 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.654298    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:40:49.654509 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:49.654307    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11dd5cb6-7a7f-42c2-9829-9abb5b70f585-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:40:50.205857 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.205821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" event={"ID":"86daa65b-d287-4f57-8e60-b63c9b8f29aa","Type":"ContainerStarted","Data":"27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f"}
Sep 17 23:40:50.206262 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.205882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" event={"ID":"86daa65b-d287-4f57-8e60-b63c9b8f29aa","Type":"ContainerStarted","Data":"6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df"}
Sep 17 23:40:50.206262 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.206067    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:50.207836 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.207811    2580 generic.go:356] "Generic (PLEG): container finished" podID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerID="0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75" exitCode=0
Sep 17 23:40:50.207953 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.207867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" event={"ID":"11dd5cb6-7a7f-42c2-9829-9abb5b70f585","Type":"ContainerDied","Data":"0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75"}
Sep 17 23:40:50.207953 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.207907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn" event={"ID":"11dd5cb6-7a7f-42c2-9829-9abb5b70f585","Type":"ContainerDied","Data":"b19df30aa2a08aa211cc71c59962e27cb40fda33df1c5f0d7c158980babc7498"}
Sep 17 23:40:50.207953 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.207934    2580 scope.go:117] "RemoveContainer" containerID="7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b"
Sep 17 23:40:50.207953 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.207868    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"
Sep 17 23:40:50.216056 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.216035    2580 scope.go:117] "RemoveContainer" containerID="0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75"
Sep 17 23:40:50.223000 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.222980    2580 scope.go:117] "RemoveContainer" containerID="e5087b4b4014cc3bea841793d92516c6b707589c161bf01ba2bf7702a9df6cc0"
Sep 17 23:40:50.226595 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.226558    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podStartSLOduration=3.266552959 podStartE2EDuration="7.226546004s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.959993045s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:40:50.225240796 +0000 UTC m=+1783.269616968" watchObservedRunningTime="2026-09-17 23:40:50.226546004 +0000 UTC m=+1783.270922164"
Sep 17 23:40:50.230065 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.230048    2580 scope.go:117] "RemoveContainer" containerID="7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b"
Sep 17 23:40:50.230347 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:40:50.230328    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b\": container with ID starting with 7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b not found: ID does not exist" containerID="7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b"
Sep 17 23:40:50.230408 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.230354    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b"} err="failed to get container status \"7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b\": rpc error: code = NotFound desc = could not find container \"7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b\": container with ID starting with 7102725aa95259feadf5515f1c22555168ce4a1161779b824f3507a36d65856b not found: ID does not exist"
Sep 17 23:40:50.230408 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.230370    2580 scope.go:117] "RemoveContainer" containerID="0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75"
Sep 17 23:40:50.230587 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:40:50.230574    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75\": container with ID starting with 0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75 not found: ID does not exist" containerID="0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75"
Sep 17 23:40:50.230652 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.230590    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75"} err="failed to get container status \"0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75\": rpc error: code = NotFound desc = could not find container \"0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75\": container with ID starting with 0e2ca626e5caa40d2b0d8c1ecc32e5d86040afb453df6d10a950172687b84c75 not found: ID does not exist"
Sep 17 23:40:50.230652 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.230601    2580 scope.go:117] "RemoveContainer" containerID="e5087b4b4014cc3bea841793d92516c6b707589c161bf01ba2bf7702a9df6cc0"
Sep 17 23:40:50.230866 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:40:50.230851    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5087b4b4014cc3bea841793d92516c6b707589c161bf01ba2bf7702a9df6cc0\": container with ID starting with e5087b4b4014cc3bea841793d92516c6b707589c161bf01ba2bf7702a9df6cc0 not found: ID does not exist" containerID="e5087b4b4014cc3bea841793d92516c6b707589c161bf01ba2bf7702a9df6cc0"
Sep 17 23:40:50.230918 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.230870    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5087b4b4014cc3bea841793d92516c6b707589c161bf01ba2bf7702a9df6cc0"} err="failed to get container status \"e5087b4b4014cc3bea841793d92516c6b707589c161bf01ba2bf7702a9df6cc0\": rpc error: code = NotFound desc = could not find container \"e5087b4b4014cc3bea841793d92516c6b707589c161bf01ba2bf7702a9df6cc0\": container with ID starting with e5087b4b4014cc3bea841793d92516c6b707589c161bf01ba2bf7702a9df6cc0 not found: ID does not exist"
Sep 17 23:40:50.237793 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.237774    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"]
Sep 17 23:40:50.242016 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:50.241999    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-rzrjn"]
Sep 17 23:40:51.213028 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:51.212999    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:51.214183 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:51.214155    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:40:51.501224 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:51.501146    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" path="/var/lib/kubelet/pods/11dd5cb6-7a7f-42c2-9829-9abb5b70f585/volumes"
Sep 17 23:40:52.220766 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.220724    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:40:52.300809 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.300774    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"]
Sep 17 23:40:52.301064 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.301051    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kube-rbac-proxy"
Sep 17 23:40:52.301107 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.301065    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kube-rbac-proxy"
Sep 17 23:40:52.301107 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.301084    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container"
Sep 17 23:40:52.301107 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.301089    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container"
Sep 17 23:40:52.301107 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.301097    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="storage-initializer"
Sep 17 23:40:52.301107 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.301102    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="storage-initializer"
Sep 17 23:40:52.301266 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.301145    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kube-rbac-proxy"
Sep 17 23:40:52.301266 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.301152    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="11dd5cb6-7a7f-42c2-9829-9abb5b70f585" containerName="kserve-container"
Sep 17 23:40:52.305129 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.305109    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.307729 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.307709    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 23:40:52.307841 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.307709    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 23:40:52.314936 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.314915    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"]
Sep 17 23:40:52.372977 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.372937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf1f904d-1d68-45d7-8829-9af855c8a28d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.373150 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.372983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf1f904d-1d68-45d7-8829-9af855c8a28d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.373150 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.373017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf1f904d-1d68-45d7-8829-9af855c8a28d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.373150 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.373044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6f8\" (UniqueName: \"kubernetes.io/projected/cf1f904d-1d68-45d7-8829-9af855c8a28d-kube-api-access-4v6f8\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.373150 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.373111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf1f904d-1d68-45d7-8829-9af855c8a28d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.474484 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.474400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf1f904d-1d68-45d7-8829-9af855c8a28d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.474484 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.474443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf1f904d-1d68-45d7-8829-9af855c8a28d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.474484 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.474473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf1f904d-1d68-45d7-8829-9af855c8a28d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.474798 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.474500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6f8\" (UniqueName: \"kubernetes.io/projected/cf1f904d-1d68-45d7-8829-9af855c8a28d-kube-api-access-4v6f8\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.474798 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.474530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf1f904d-1d68-45d7-8829-9af855c8a28d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.475040 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.475019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf1f904d-1d68-45d7-8829-9af855c8a28d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.475237 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.475215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf1f904d-1d68-45d7-8829-9af855c8a28d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.475281 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.475215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf1f904d-1d68-45d7-8829-9af855c8a28d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.477134 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.477113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf1f904d-1d68-45d7-8829-9af855c8a28d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.483153 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.483132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6f8\" (UniqueName: \"kubernetes.io/projected/cf1f904d-1d68-45d7-8829-9af855c8a28d-kube-api-access-4v6f8\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-rgg2x\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.615407 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.615372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:52.890038 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:52.890006    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"]
Sep 17 23:40:52.892374 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:40:52.892349    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1f904d_1d68_45d7_8829_9af855c8a28d.slice/crio-3bf26384d30af6d22fab470b0442c3129c5ba250b0a8f354fa99ab9e453f0b50 WatchSource:0}: Error finding container 3bf26384d30af6d22fab470b0442c3129c5ba250b0a8f354fa99ab9e453f0b50: Status 404 returned error can't find the container with id 3bf26384d30af6d22fab470b0442c3129c5ba250b0a8f354fa99ab9e453f0b50
Sep 17 23:40:53.222504 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:53.222409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" event={"ID":"cf1f904d-1d68-45d7-8829-9af855c8a28d","Type":"ContainerStarted","Data":"f82b7e6292e2c1a71a47a3c502c064acc72eef68ec2c4f2e1b1c18db0b173560"}
Sep 17 23:40:53.222504 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:53.222485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" event={"ID":"cf1f904d-1d68-45d7-8829-9af855c8a28d","Type":"ContainerStarted","Data":"3bf26384d30af6d22fab470b0442c3129c5ba250b0a8f354fa99ab9e453f0b50"}
Sep 17 23:40:57.222703 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:57.222602    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:40:57.223095 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:57.223071    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:40:57.238513 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:57.238489    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerID="f82b7e6292e2c1a71a47a3c502c064acc72eef68ec2c4f2e1b1c18db0b173560" exitCode=0
Sep 17 23:40:57.238657 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:57.238562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" event={"ID":"cf1f904d-1d68-45d7-8829-9af855c8a28d","Type":"ContainerDied","Data":"f82b7e6292e2c1a71a47a3c502c064acc72eef68ec2c4f2e1b1c18db0b173560"}
Sep 17 23:40:58.244520 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:58.244485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" event={"ID":"cf1f904d-1d68-45d7-8829-9af855c8a28d","Type":"ContainerStarted","Data":"ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75"}
Sep 17 23:40:58.244948 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:58.244573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" event={"ID":"cf1f904d-1d68-45d7-8829-9af855c8a28d","Type":"ContainerStarted","Data":"9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa"}
Sep 17 23:40:58.244948 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:58.244909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:58.245064 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:58.245045    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:40:58.246224 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:58.246200    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:40:58.265912 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:58.265866    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podStartSLOduration=2.349802822 podStartE2EDuration="6.265852007s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.916049185s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:40:58.264471712 +0000 UTC m=+1791.308847874" watchObservedRunningTime="2026-09-17 23:40:58.265852007 +0000 UTC m=+1791.310228168"
Sep 17 23:40:59.248718 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:40:59.248639    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:41:04.253275 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:04.253245    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:41:04.253804 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:04.253697    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:41:07.223129 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:07.223089    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:41:14.253824 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:14.253786    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:41:17.223420 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:17.223381    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:41:24.253961 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:24.253924    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:41:27.223942 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:27.223905    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:41:34.254592 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:34.254552    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:41:37.223756 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:37.223716    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:41:44.253736 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:44.253693    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:41:47.223258 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:47.223217    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:41:54.254572 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:54.254533    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:41:57.223051 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:41:57.223011    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:42:04.253797 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:04.253756    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:42:07.224130 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:07.224096    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:42:13.783813 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:13.783780    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"]
Sep 17 23:42:13.784333 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:13.784029    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container" containerID="cri-o://6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df" gracePeriod=30
Sep 17 23:42:13.784333 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:13.784068    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kube-rbac-proxy" containerID="cri-o://27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f" gracePeriod=30
Sep 17 23:42:14.254892 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:14.254806    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:42:14.535977 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:14.535941    2580 generic.go:356] "Generic (PLEG): container finished" podID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerID="27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f" exitCode=2
Sep 17 23:42:14.536151 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:14.535990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" event={"ID":"86daa65b-d287-4f57-8e60-b63c9b8f29aa","Type":"ContainerDied","Data":"27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f"}
Sep 17 23:42:17.219461 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:17.219413    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 23:42:17.223773 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:17.223742    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:42:18.523269 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.523244    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:42:18.552159 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.552128    2580 generic.go:356] "Generic (PLEG): container finished" podID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerID="6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df" exitCode=0
Sep 17 23:42:18.552323 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.552173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" event={"ID":"86daa65b-d287-4f57-8e60-b63c9b8f29aa","Type":"ContainerDied","Data":"6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df"}
Sep 17 23:42:18.552323 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.552186    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"
Sep 17 23:42:18.552323 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.552247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2" event={"ID":"86daa65b-d287-4f57-8e60-b63c9b8f29aa","Type":"ContainerDied","Data":"89ade21a7d5f70adff3897f89aedef5e5c8b3658ec3533dfa3dd5bbfa2e95772"}
Sep 17 23:42:18.552323 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.552277    2580 scope.go:117] "RemoveContainer" containerID="27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f"
Sep 17 23:42:18.562532 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.562513    2580 scope.go:117] "RemoveContainer" containerID="6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df"
Sep 17 23:42:18.569934 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.569915    2580 scope.go:117] "RemoveContainer" containerID="cd455e1749799fe4bc9748201080d1267b2c67532e579e76cd952799c8438b1c"
Sep 17 23:42:18.576932 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.576905    2580 scope.go:117] "RemoveContainer" containerID="27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f"
Sep 17 23:42:18.577203 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:42:18.577184    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f\": container with ID starting with 27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f not found: ID does not exist" containerID="27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f"
Sep 17 23:42:18.577279 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.577217    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f"} err="failed to get container status \"27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f\": rpc error: code = NotFound desc = could not find container \"27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f\": container with ID starting with 27a85582d6e6b784c0769af8fc9d75861a3962b40b395e9aa9fb8450ff089d8f not found: ID does not exist"
Sep 17 23:42:18.577279 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.577243    2580 scope.go:117] "RemoveContainer" containerID="6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df"
Sep 17 23:42:18.577511 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:42:18.577493    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df\": container with ID starting with 6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df not found: ID does not exist" containerID="6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df"
Sep 17 23:42:18.577550 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.577521    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df"} err="failed to get container status \"6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df\": rpc error: code = NotFound desc = could not find container \"6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df\": container with ID starting with 6fbe370cd9839178bd96a76c94bf8222029fb8daab3c329dbb51e0f3944db1df not found: ID does not exist"
Sep 17 23:42:18.577550 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.577539    2580 scope.go:117] "RemoveContainer" containerID="cd455e1749799fe4bc9748201080d1267b2c67532e579e76cd952799c8438b1c"
Sep 17 23:42:18.577785 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:42:18.577768    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd455e1749799fe4bc9748201080d1267b2c67532e579e76cd952799c8438b1c\": container with ID starting with cd455e1749799fe4bc9748201080d1267b2c67532e579e76cd952799c8438b1c not found: ID does not exist" containerID="cd455e1749799fe4bc9748201080d1267b2c67532e579e76cd952799c8438b1c"
Sep 17 23:42:18.577835 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.577789    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd455e1749799fe4bc9748201080d1267b2c67532e579e76cd952799c8438b1c"} err="failed to get container status \"cd455e1749799fe4bc9748201080d1267b2c67532e579e76cd952799c8438b1c\": rpc error: code = NotFound desc = could not find container \"cd455e1749799fe4bc9748201080d1267b2c67532e579e76cd952799c8438b1c\": container with ID starting with cd455e1749799fe4bc9748201080d1267b2c67532e579e76cd952799c8438b1c not found: ID does not exist"
Sep 17 23:42:18.618576 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.618554    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kube-rbac-proxy-sar-config\") pod \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") "
Sep 17 23:42:18.618690 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.618652    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kserve-provision-location\") pod \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") "
Sep 17 23:42:18.618690 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.618676    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7ttd\" (UniqueName: \"kubernetes.io/projected/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kube-api-access-c7ttd\") pod \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") "
Sep 17 23:42:18.618765 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.618714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86daa65b-d287-4f57-8e60-b63c9b8f29aa-proxy-tls\") pod \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") "
Sep 17 23:42:18.618765 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.618747    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86daa65b-d287-4f57-8e60-b63c9b8f29aa-cabundle-cert\") pod \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\" (UID: \"86daa65b-d287-4f57-8e60-b63c9b8f29aa\") "
Sep 17 23:42:18.618953 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.618924    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86daa65b-d287-4f57-8e60-b63c9b8f29aa" (UID: "86daa65b-d287-4f57-8e60-b63c9b8f29aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:42:18.619023 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.618949    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86daa65b-d287-4f57-8e60-b63c9b8f29aa" (UID: "86daa65b-d287-4f57-8e60-b63c9b8f29aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:42:18.619147 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.619126    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86daa65b-d287-4f57-8e60-b63c9b8f29aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86daa65b-d287-4f57-8e60-b63c9b8f29aa" (UID: "86daa65b-d287-4f57-8e60-b63c9b8f29aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:42:18.620826 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.620801    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kube-api-access-c7ttd" (OuterVolumeSpecName: "kube-api-access-c7ttd") pod "86daa65b-d287-4f57-8e60-b63c9b8f29aa" (UID: "86daa65b-d287-4f57-8e60-b63c9b8f29aa"). InnerVolumeSpecName "kube-api-access-c7ttd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:42:18.620919 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.620857    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86daa65b-d287-4f57-8e60-b63c9b8f29aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86daa65b-d287-4f57-8e60-b63c9b8f29aa" (UID: "86daa65b-d287-4f57-8e60-b63c9b8f29aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:42:18.719386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.719346    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:42:18.719386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.719377    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7ttd\" (UniqueName: \"kubernetes.io/projected/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kube-api-access-c7ttd\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:42:18.719386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.719387    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86daa65b-d287-4f57-8e60-b63c9b8f29aa-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:42:18.719386 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.719398    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86daa65b-d287-4f57-8e60-b63c9b8f29aa-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:42:18.719688 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.719406    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86daa65b-d287-4f57-8e60-b63c9b8f29aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:42:18.874110 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.874083    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"]
Sep 17 23:42:18.880591 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:18.880530    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-745c694d54-ptzd2"]
Sep 17 23:42:19.500930 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:19.500899    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" path="/var/lib/kubelet/pods/86daa65b-d287-4f57-8e60-b63c9b8f29aa/volumes"
Sep 17 23:42:22.439884 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:22.439848    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"]
Sep 17 23:42:22.440319 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:22.440085    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container" containerID="cri-o://9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa" gracePeriod=30
Sep 17 23:42:22.440319 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:22.440136    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kube-rbac-proxy" containerID="cri-o://ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75" gracePeriod=30
Sep 17 23:42:22.567459 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:22.567426    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerID="ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75" exitCode=2
Sep 17 23:42:22.567586 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:22.567500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" event={"ID":"cf1f904d-1d68-45d7-8829-9af855c8a28d","Type":"ContainerDied","Data":"ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75"}
Sep 17 23:42:24.249210 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:24.249168    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 23:42:24.253748 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:24.253721    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:42:27.182630 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.182594    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:42:27.286707 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.286684    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf1f904d-1d68-45d7-8829-9af855c8a28d-kube-rbac-proxy-sar-config\") pod \"cf1f904d-1d68-45d7-8829-9af855c8a28d\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") "
Sep 17 23:42:27.286878 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.286735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf1f904d-1d68-45d7-8829-9af855c8a28d-proxy-tls\") pod \"cf1f904d-1d68-45d7-8829-9af855c8a28d\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") "
Sep 17 23:42:27.286878 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.286757    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v6f8\" (UniqueName: \"kubernetes.io/projected/cf1f904d-1d68-45d7-8829-9af855c8a28d-kube-api-access-4v6f8\") pod \"cf1f904d-1d68-45d7-8829-9af855c8a28d\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") "
Sep 17 23:42:27.286878 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.286793    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf1f904d-1d68-45d7-8829-9af855c8a28d-cabundle-cert\") pod \"cf1f904d-1d68-45d7-8829-9af855c8a28d\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") "
Sep 17 23:42:27.286878 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.286834    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf1f904d-1d68-45d7-8829-9af855c8a28d-kserve-provision-location\") pod \"cf1f904d-1d68-45d7-8829-9af855c8a28d\" (UID: \"cf1f904d-1d68-45d7-8829-9af855c8a28d\") "
Sep 17 23:42:27.287165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.287133    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf1f904d-1d68-45d7-8829-9af855c8a28d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf1f904d-1d68-45d7-8829-9af855c8a28d" (UID: "cf1f904d-1d68-45d7-8829-9af855c8a28d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:42:27.287289 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.287224    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf1f904d-1d68-45d7-8829-9af855c8a28d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf1f904d-1d68-45d7-8829-9af855c8a28d" (UID: "cf1f904d-1d68-45d7-8829-9af855c8a28d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:42:27.287289 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.287221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1f904d-1d68-45d7-8829-9af855c8a28d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf1f904d-1d68-45d7-8829-9af855c8a28d" (UID: "cf1f904d-1d68-45d7-8829-9af855c8a28d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:42:27.289025 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.288999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1f904d-1d68-45d7-8829-9af855c8a28d-kube-api-access-4v6f8" (OuterVolumeSpecName: "kube-api-access-4v6f8") pod "cf1f904d-1d68-45d7-8829-9af855c8a28d" (UID: "cf1f904d-1d68-45d7-8829-9af855c8a28d"). InnerVolumeSpecName "kube-api-access-4v6f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:42:27.289100 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.289069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf1f904d-1d68-45d7-8829-9af855c8a28d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf1f904d-1d68-45d7-8829-9af855c8a28d" (UID: "cf1f904d-1d68-45d7-8829-9af855c8a28d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:42:27.387812 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.387768    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf1f904d-1d68-45d7-8829-9af855c8a28d-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:42:27.387812 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.387807    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v6f8\" (UniqueName: \"kubernetes.io/projected/cf1f904d-1d68-45d7-8829-9af855c8a28d-kube-api-access-4v6f8\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:42:27.387812 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.387819    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf1f904d-1d68-45d7-8829-9af855c8a28d-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:42:27.387812 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.387828    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf1f904d-1d68-45d7-8829-9af855c8a28d-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:42:27.388094 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.387836    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf1f904d-1d68-45d7-8829-9af855c8a28d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:42:27.589725 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.589637    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerID="9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa" exitCode=0
Sep 17 23:42:27.589725 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.589707    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"
Sep 17 23:42:27.589948 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.589726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" event={"ID":"cf1f904d-1d68-45d7-8829-9af855c8a28d","Type":"ContainerDied","Data":"9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa"}
Sep 17 23:42:27.589948 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.589797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x" event={"ID":"cf1f904d-1d68-45d7-8829-9af855c8a28d","Type":"ContainerDied","Data":"3bf26384d30af6d22fab470b0442c3129c5ba250b0a8f354fa99ab9e453f0b50"}
Sep 17 23:42:27.589948 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.589835    2580 scope.go:117] "RemoveContainer" containerID="ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75"
Sep 17 23:42:27.597855 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.597831    2580 scope.go:117] "RemoveContainer" containerID="9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa"
Sep 17 23:42:27.606516 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.606493    2580 scope.go:117] "RemoveContainer" containerID="f82b7e6292e2c1a71a47a3c502c064acc72eef68ec2c4f2e1b1c18db0b173560"
Sep 17 23:42:27.613577 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.613551    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"]
Sep 17 23:42:27.614972 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.614958    2580 scope.go:117] "RemoveContainer" containerID="ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75"
Sep 17 23:42:27.615282 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:42:27.615257    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75\": container with ID starting with ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75 not found: ID does not exist" containerID="ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75"
Sep 17 23:42:27.615337 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615293    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75"} err="failed to get container status \"ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75\": rpc error: code = NotFound desc = could not find container \"ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75\": container with ID starting with ed83be9a0f2c88d208363bfec629ce3880fc14297fcb8a2243018c98346c2f75 not found: ID does not exist"
Sep 17 23:42:27.615337 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615312    2580 scope.go:117] "RemoveContainer" containerID="9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa"
Sep 17 23:42:27.615430 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615356    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"]
Sep 17 23:42:27.615562 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:42:27.615544    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa\": container with ID starting with 9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa not found: ID does not exist" containerID="9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa"
Sep 17 23:42:27.615647 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615571    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa"} err="failed to get container status \"9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa\": rpc error: code = NotFound desc = could not find container \"9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa\": container with ID starting with 9ead164f940d96a7d2769eba04b7a130bf70089bd7605e7eff52df398036ecaa not found: ID does not exist"
Sep 17 23:42:27.615647 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615593    2580 scope.go:117] "RemoveContainer" containerID="f82b7e6292e2c1a71a47a3c502c064acc72eef68ec2c4f2e1b1c18db0b173560"
Sep 17 23:42:27.615647 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615637    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container"
Sep 17 23:42:27.615647 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615649    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615656    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kube-rbac-proxy"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615662    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kube-rbac-proxy"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615671    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="storage-initializer"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615676    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="storage-initializer"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615691    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615696    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615702    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kube-rbac-proxy"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615709    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kube-rbac-proxy"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615741    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="storage-initializer"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615747    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="storage-initializer"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615791    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kserve-container"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615798    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kserve-container"
Sep 17 23:42:27.615808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615809    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="86daa65b-d287-4f57-8e60-b63c9b8f29aa" containerName="kube-rbac-proxy"
Sep 17 23:42:27.616224 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615816    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" containerName="kube-rbac-proxy"
Sep 17 23:42:27.616224 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:42:27.615867    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82b7e6292e2c1a71a47a3c502c064acc72eef68ec2c4f2e1b1c18db0b173560\": container with ID starting with f82b7e6292e2c1a71a47a3c502c064acc72eef68ec2c4f2e1b1c18db0b173560 not found: ID does not exist" containerID="f82b7e6292e2c1a71a47a3c502c064acc72eef68ec2c4f2e1b1c18db0b173560"
Sep 17 23:42:27.616224 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.615892    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82b7e6292e2c1a71a47a3c502c064acc72eef68ec2c4f2e1b1c18db0b173560"} err="failed to get container status \"f82b7e6292e2c1a71a47a3c502c064acc72eef68ec2c4f2e1b1c18db0b173560\": rpc error: code = NotFound desc = could not find container \"f82b7e6292e2c1a71a47a3c502c064acc72eef68ec2c4f2e1b1c18db0b173560\": container with ID starting with f82b7e6292e2c1a71a47a3c502c064acc72eef68ec2c4f2e1b1c18db0b173560 not found: ID does not exist"
Sep 17 23:42:27.620047 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.620030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.620551 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.620532    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-rgg2x"]
Sep 17 23:42:27.622337 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.622270    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 23:42:27.622337 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.622316    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:42:27.622337 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.622325    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 23:42:27.622536 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.622337    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:42:27.622536 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.622349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:42:27.622536 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.622317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:42:27.622772 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.622758    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:42:27.629322 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.629298    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"]
Sep 17 23:42:27.655044 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.655015    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"]
Sep 17 23:42:27.658651 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.658634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.660907 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.660887    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 23:42:27.666889 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.666866    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"]
Sep 17 23:42:27.790857 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.790819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4230922d-d05b-40ad-9f23-a5dcc584ae53-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.790857 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.790861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjv52\" (UniqueName: \"kubernetes.io/projected/4230922d-d05b-40ad-9f23-a5dcc584ae53-kube-api-access-kjv52\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.791136 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.790887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqspm\" (UniqueName: \"kubernetes.io/projected/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kube-api-access-qqspm\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.791136 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.790917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4230922d-d05b-40ad-9f23-a5dcc584ae53-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.791136 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.790943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bf64c5b-051c-482c-9a27-cd322ea6ae01-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.791136 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.790962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4230922d-d05b-40ad-9f23-a5dcc584ae53-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.791136 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.790994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.791136 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.791009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.791136 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.791023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bf64c5b-051c-482c-9a27-cd322ea6ae01-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.791136 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.791039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4230922d-d05b-40ad-9f23-a5dcc584ae53-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.892024 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.891927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqspm\" (UniqueName: \"kubernetes.io/projected/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kube-api-access-qqspm\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.892024 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.891976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4230922d-d05b-40ad-9f23-a5dcc584ae53-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.892211 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.892149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bf64c5b-051c-482c-9a27-cd322ea6ae01-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.892211 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.892193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4230922d-d05b-40ad-9f23-a5dcc584ae53-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.892276 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.892225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.892276 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.892249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.892385 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.892272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bf64c5b-051c-482c-9a27-cd322ea6ae01-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.892385 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.892289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4230922d-d05b-40ad-9f23-a5dcc584ae53-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.892385 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.892300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4230922d-d05b-40ad-9f23-a5dcc584ae53-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.892385 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.892369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4230922d-d05b-40ad-9f23-a5dcc584ae53-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.892590 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.892409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjv52\" (UniqueName: \"kubernetes.io/projected/4230922d-d05b-40ad-9f23-a5dcc584ae53-kube-api-access-kjv52\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.892697 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.892668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.892894 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.892866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4230922d-d05b-40ad-9f23-a5dcc584ae53-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.893044 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.893021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.893362 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.893319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bf64c5b-051c-482c-9a27-cd322ea6ae01-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.893453 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.893433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4230922d-d05b-40ad-9f23-a5dcc584ae53-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.894881 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.894857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4230922d-d05b-40ad-9f23-a5dcc584ae53-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.894966 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.894932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bf64c5b-051c-482c-9a27-cd322ea6ae01-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.901054 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.901030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqspm\" (UniqueName: \"kubernetes.io/projected/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kube-api-access-qqspm\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.901233 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.901214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjv52\" (UniqueName: \"kubernetes.io/projected/4230922d-d05b-40ad-9f23-a5dcc584ae53-kube-api-access-kjv52\") pod \"isvc-canary-rollback-predictor-7df6589c46-525lx\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:27.930588 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.930559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:27.969651 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:27.969593    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:28.202523 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:28.202433    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"]
Sep 17 23:42:28.205463 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:42:28.205430    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf64c5b_051c_482c_9a27_cd322ea6ae01.slice/crio-8beff92f99e2c284246fe60179cafb3814c92a3679e6419c487abf293574d1d3 WatchSource:0}: Error finding container 8beff92f99e2c284246fe60179cafb3814c92a3679e6419c487abf293574d1d3: Status 404 returned error can't find the container with id 8beff92f99e2c284246fe60179cafb3814c92a3679e6419c487abf293574d1d3
Sep 17 23:42:28.247680 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:28.247651    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"]
Sep 17 23:42:28.250494 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:42:28.250469    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4230922d_d05b_40ad_9f23_a5dcc584ae53.slice/crio-50efa409d75c9ad92e659e71a177469be57722a772fc3d0277dd68d45adba469 WatchSource:0}: Error finding container 50efa409d75c9ad92e659e71a177469be57722a772fc3d0277dd68d45adba469: Status 404 returned error can't find the container with id 50efa409d75c9ad92e659e71a177469be57722a772fc3d0277dd68d45adba469
Sep 17 23:42:28.596324 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:28.596282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" event={"ID":"4230922d-d05b-40ad-9f23-a5dcc584ae53","Type":"ContainerStarted","Data":"2497a9ee628af8c7926f5fc4916b0f146e1ce0860fdcb6da0a1afb6c655d060d"}
Sep 17 23:42:28.596672 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:28.596344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" event={"ID":"4230922d-d05b-40ad-9f23-a5dcc584ae53","Type":"ContainerStarted","Data":"50efa409d75c9ad92e659e71a177469be57722a772fc3d0277dd68d45adba469"}
Sep 17 23:42:28.598983 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:28.598958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" event={"ID":"2bf64c5b-051c-482c-9a27-cd322ea6ae01","Type":"ContainerStarted","Data":"261d5da83bdef676afe368f76ecc432c83861327a688609f42a4fe24696b2282"}
Sep 17 23:42:28.599098 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:28.599004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" event={"ID":"2bf64c5b-051c-482c-9a27-cd322ea6ae01","Type":"ContainerStarted","Data":"8beff92f99e2c284246fe60179cafb3814c92a3679e6419c487abf293574d1d3"}
Sep 17 23:42:29.500586 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:29.500554    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf1f904d-1d68-45d7-8829-9af855c8a28d" path="/var/lib/kubelet/pods/cf1f904d-1d68-45d7-8829-9af855c8a28d/volumes"
Sep 17 23:42:32.616268 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:32.616231    2580 generic.go:356] "Generic (PLEG): container finished" podID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerID="2497a9ee628af8c7926f5fc4916b0f146e1ce0860fdcb6da0a1afb6c655d060d" exitCode=0
Sep 17 23:42:32.616700 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:32.616310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" event={"ID":"4230922d-d05b-40ad-9f23-a5dcc584ae53","Type":"ContainerDied","Data":"2497a9ee628af8c7926f5fc4916b0f146e1ce0860fdcb6da0a1afb6c655d060d"}
Sep 17 23:42:32.618035 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:32.618012    2580 generic.go:356] "Generic (PLEG): container finished" podID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerID="261d5da83bdef676afe368f76ecc432c83861327a688609f42a4fe24696b2282" exitCode=0
Sep 17 23:42:32.618124 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:32.618059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" event={"ID":"2bf64c5b-051c-482c-9a27-cd322ea6ae01","Type":"ContainerDied","Data":"261d5da83bdef676afe368f76ecc432c83861327a688609f42a4fe24696b2282"}
Sep 17 23:42:33.624099 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:33.624062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" event={"ID":"2bf64c5b-051c-482c-9a27-cd322ea6ae01","Type":"ContainerStarted","Data":"d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f"}
Sep 17 23:42:33.624501 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:33.624128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" event={"ID":"2bf64c5b-051c-482c-9a27-cd322ea6ae01","Type":"ContainerStarted","Data":"22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf"}
Sep 17 23:42:33.624501 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:33.624375    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:33.626152 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:33.626131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" event={"ID":"4230922d-d05b-40ad-9f23-a5dcc584ae53","Type":"ContainerStarted","Data":"74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be"}
Sep 17 23:42:33.626248 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:33.626175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" event={"ID":"4230922d-d05b-40ad-9f23-a5dcc584ae53","Type":"ContainerStarted","Data":"f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f"}
Sep 17 23:42:33.626485 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:33.626469    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:33.626542 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:33.626520    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:33.627749 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:33.627726    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 23:42:33.644657 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:33.644592    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podStartSLOduration=2.880921844 podStartE2EDuration="6.644579992s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.763658148s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:42:33.643118097 +0000 UTC m=+1886.687494258" watchObservedRunningTime="2026-09-17 23:42:33.644579992 +0000 UTC m=+1886.688956209"
Sep 17 23:42:33.665956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:33.665914    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podStartSLOduration=2.862549847 podStartE2EDuration="6.665899026s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.803349179s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:42:33.664370546 +0000 UTC m=+1886.708746701" watchObservedRunningTime="2026-09-17 23:42:33.665899026 +0000 UTC m=+1886.710275187"
Sep 17 23:42:34.630674 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:34.630641    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:34.631071 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:34.630666    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 23:42:34.632064 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:34.632042    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:42:35.635365 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:35.635322    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:42:39.635284 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:39.635251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:42:39.635893 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:39.635865    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 23:42:40.640219 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:40.640192    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:42:40.640710 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:40.640684    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:42:49.636710 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:49.636666    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 23:42:50.641315 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:50.641279    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:42:59.636284 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:42:59.636244    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 23:43:00.641541 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:00.641503    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:43:09.636534 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:09.636489    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 23:43:10.641047 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:10.641005    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:43:19.636520 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:19.636480    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 23:43:20.641129 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:20.641087    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:43:29.636260 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:29.636210    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 23:43:30.641297 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:30.641258    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:43:39.636385 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:39.636342    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 23:43:40.640798 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:40.640756    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:43:42.496568 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:42.496519    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 23:43:47.614927 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.614889    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"]
Sep 17 23:43:47.618189 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.618171    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.620724 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.620700    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 17 23:43:47.620841 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.620736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 17 23:43:47.628056 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.628034    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"]
Sep 17 23:43:47.702978 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.702945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.702978 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.702978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.703221 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.703013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40368dd0-36f7-4e45-8aa3-20103cca9dfd-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.703221 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.703108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40368dd0-36f7-4e45-8aa3-20103cca9dfd-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.703221 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.703141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txr2s\" (UniqueName: \"kubernetes.io/projected/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kube-api-access-txr2s\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.803969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.803939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40368dd0-36f7-4e45-8aa3-20103cca9dfd-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.803969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.803971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txr2s\" (UniqueName: \"kubernetes.io/projected/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kube-api-access-txr2s\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.804153 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.803992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.804153 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.804120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.804261 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.804175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40368dd0-36f7-4e45-8aa3-20103cca9dfd-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.804471 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.804455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.804713 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.804691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40368dd0-36f7-4e45-8aa3-20103cca9dfd-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.804843 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.804826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.806862 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.806843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40368dd0-36f7-4e45-8aa3-20103cca9dfd-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.813093 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.813072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txr2s\" (UniqueName: \"kubernetes.io/projected/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kube-api-access-txr2s\") pod \"isvc-triton-predictor-5d7db58676-frcfj\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:47.928928 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:47.928842    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:43:48.200792 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:48.200710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"]
Sep 17 23:43:48.203601 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:43:48.203572    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40368dd0_36f7_4e45_8aa3_20103cca9dfd.slice/crio-b48bab6b0e2471e54f3c2c7e3d2260750b35745e1ae4f4bfd16790e5a1253b28 WatchSource:0}: Error finding container b48bab6b0e2471e54f3c2c7e3d2260750b35745e1ae4f4bfd16790e5a1253b28: Status 404 returned error can't find the container with id b48bab6b0e2471e54f3c2c7e3d2260750b35745e1ae4f4bfd16790e5a1253b28
Sep 17 23:43:48.898407 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:48.898374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" event={"ID":"40368dd0-36f7-4e45-8aa3-20103cca9dfd","Type":"ContainerStarted","Data":"a9b1a91201cd9ef6ce39e25e5cc59d5dc26a7678cc3ca0df8fc32a669103fd82"}
Sep 17 23:43:48.898814 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:48.898431    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" event={"ID":"40368dd0-36f7-4e45-8aa3-20103cca9dfd","Type":"ContainerStarted","Data":"b48bab6b0e2471e54f3c2c7e3d2260750b35745e1ae4f4bfd16790e5a1253b28"}
Sep 17 23:43:50.641344 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:50.641318    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:43:50.754876 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:50.754841    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"]
Sep 17 23:43:50.755114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:50.755091    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container" containerID="cri-o://f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f" gracePeriod=30
Sep 17 23:43:50.755219 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:50.755167    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kube-rbac-proxy" containerID="cri-o://74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be" gracePeriod=30
Sep 17 23:43:50.755939 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:50.755789    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:43:50.907436 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:50.907365    2580 generic.go:356] "Generic (PLEG): container finished" podID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerID="74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be" exitCode=2
Sep 17 23:43:50.907563 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:50.907434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" event={"ID":"4230922d-d05b-40ad-9f23-a5dcc584ae53","Type":"ContainerDied","Data":"74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be"}
Sep 17 23:43:52.916361 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:52.916332    2580 generic.go:356] "Generic (PLEG): container finished" podID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerID="a9b1a91201cd9ef6ce39e25e5cc59d5dc26a7678cc3ca0df8fc32a669103fd82" exitCode=0
Sep 17 23:43:52.916819 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:52.916413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" event={"ID":"40368dd0-36f7-4e45-8aa3-20103cca9dfd","Type":"ContainerDied","Data":"a9b1a91201cd9ef6ce39e25e5cc59d5dc26a7678cc3ca0df8fc32a669103fd82"}
Sep 17 23:43:52.917543 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:52.917526    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:43:54.632185 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:54.631876    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 17 23:43:56.438605 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.438461    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:43:56.487102 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.487011    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjv52\" (UniqueName: \"kubernetes.io/projected/4230922d-d05b-40ad-9f23-a5dcc584ae53-kube-api-access-kjv52\") pod \"4230922d-d05b-40ad-9f23-a5dcc584ae53\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") "
Sep 17 23:43:56.487102 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.487057    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4230922d-d05b-40ad-9f23-a5dcc584ae53-kube-rbac-proxy-sar-config\") pod \"4230922d-d05b-40ad-9f23-a5dcc584ae53\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") "
Sep 17 23:43:56.487353 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.487103    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4230922d-d05b-40ad-9f23-a5dcc584ae53-proxy-tls\") pod \"4230922d-d05b-40ad-9f23-a5dcc584ae53\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") "
Sep 17 23:43:56.487353 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.487147    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4230922d-d05b-40ad-9f23-a5dcc584ae53-kserve-provision-location\") pod \"4230922d-d05b-40ad-9f23-a5dcc584ae53\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") "
Sep 17 23:43:56.487353 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.487183    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4230922d-d05b-40ad-9f23-a5dcc584ae53-cabundle-cert\") pod \"4230922d-d05b-40ad-9f23-a5dcc584ae53\" (UID: \"4230922d-d05b-40ad-9f23-a5dcc584ae53\") "
Sep 17 23:43:56.488728 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.488696    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4230922d-d05b-40ad-9f23-a5dcc584ae53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4230922d-d05b-40ad-9f23-a5dcc584ae53" (UID: "4230922d-d05b-40ad-9f23-a5dcc584ae53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:43:56.488894 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.488874    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4230922d-d05b-40ad-9f23-a5dcc584ae53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4230922d-d05b-40ad-9f23-a5dcc584ae53" (UID: "4230922d-d05b-40ad-9f23-a5dcc584ae53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:43:56.489113 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.489075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4230922d-d05b-40ad-9f23-a5dcc584ae53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4230922d-d05b-40ad-9f23-a5dcc584ae53" (UID: "4230922d-d05b-40ad-9f23-a5dcc584ae53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:43:56.494230 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.494159    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4230922d-d05b-40ad-9f23-a5dcc584ae53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4230922d-d05b-40ad-9f23-a5dcc584ae53" (UID: "4230922d-d05b-40ad-9f23-a5dcc584ae53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:43:56.495602 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.495457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4230922d-d05b-40ad-9f23-a5dcc584ae53-kube-api-access-kjv52" (OuterVolumeSpecName: "kube-api-access-kjv52") pod "4230922d-d05b-40ad-9f23-a5dcc584ae53" (UID: "4230922d-d05b-40ad-9f23-a5dcc584ae53"). InnerVolumeSpecName "kube-api-access-kjv52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:43:56.587934 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.587882    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjv52\" (UniqueName: \"kubernetes.io/projected/4230922d-d05b-40ad-9f23-a5dcc584ae53-kube-api-access-kjv52\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:43:56.587934 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.587923    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4230922d-d05b-40ad-9f23-a5dcc584ae53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:43:56.587934 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.587939    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4230922d-d05b-40ad-9f23-a5dcc584ae53-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:43:56.588251 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.587954    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4230922d-d05b-40ad-9f23-a5dcc584ae53-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:43:56.588251 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.587969    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4230922d-d05b-40ad-9f23-a5dcc584ae53-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:43:56.946805 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.944350    2580 generic.go:356] "Generic (PLEG): container finished" podID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerID="f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f" exitCode=0
Sep 17 23:43:56.946805 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.944462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" event={"ID":"4230922d-d05b-40ad-9f23-a5dcc584ae53","Type":"ContainerDied","Data":"f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f"}
Sep 17 23:43:56.946805 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.944527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx" event={"ID":"4230922d-d05b-40ad-9f23-a5dcc584ae53","Type":"ContainerDied","Data":"50efa409d75c9ad92e659e71a177469be57722a772fc3d0277dd68d45adba469"}
Sep 17 23:43:56.946805 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.944565    2580 scope.go:117] "RemoveContainer" containerID="74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be"
Sep 17 23:43:56.946805 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.944764    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"
Sep 17 23:43:56.970383 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.970356    2580 scope.go:117] "RemoveContainer" containerID="f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f"
Sep 17 23:43:56.980176 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.980126    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"]
Sep 17 23:43:56.983116 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.983090    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-525lx"]
Sep 17 23:43:56.990811 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:56.990792    2580 scope.go:117] "RemoveContainer" containerID="2497a9ee628af8c7926f5fc4916b0f146e1ce0860fdcb6da0a1afb6c655d060d"
Sep 17 23:43:57.010079 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.009949    2580 scope.go:117] "RemoveContainer" containerID="74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be"
Sep 17 23:43:57.012078 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:43:57.011888    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be\": container with ID starting with 74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be not found: ID does not exist" containerID="74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be"
Sep 17 23:43:57.012078 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.011926    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be"} err="failed to get container status \"74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be\": rpc error: code = NotFound desc = could not find container \"74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be\": container with ID starting with 74d4b20d4af64b48e9c5beda570ac523a8f90ea9cf249faee956270caa4a05be not found: ID does not exist"
Sep 17 23:43:57.012078 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.011953    2580 scope.go:117] "RemoveContainer" containerID="f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f"
Sep 17 23:43:57.012823 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:43:57.012794    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f\": container with ID starting with f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f not found: ID does not exist" containerID="f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f"
Sep 17 23:43:57.012914 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.012834    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f"} err="failed to get container status \"f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f\": rpc error: code = NotFound desc = could not find container \"f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f\": container with ID starting with f6e8a74fe478ad775039780caf22108993c069ffd7237e8232e16e64da3b9f7f not found: ID does not exist"
Sep 17 23:43:57.012914 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.012858    2580 scope.go:117] "RemoveContainer" containerID="2497a9ee628af8c7926f5fc4916b0f146e1ce0860fdcb6da0a1afb6c655d060d"
Sep 17 23:43:57.013939 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:43:57.013903    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2497a9ee628af8c7926f5fc4916b0f146e1ce0860fdcb6da0a1afb6c655d060d\": container with ID starting with 2497a9ee628af8c7926f5fc4916b0f146e1ce0860fdcb6da0a1afb6c655d060d not found: ID does not exist" containerID="2497a9ee628af8c7926f5fc4916b0f146e1ce0860fdcb6da0a1afb6c655d060d"
Sep 17 23:43:57.014038 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.013938    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2497a9ee628af8c7926f5fc4916b0f146e1ce0860fdcb6da0a1afb6c655d060d"} err="failed to get container status \"2497a9ee628af8c7926f5fc4916b0f146e1ce0860fdcb6da0a1afb6c655d060d\": rpc error: code = NotFound desc = could not find container \"2497a9ee628af8c7926f5fc4916b0f146e1ce0860fdcb6da0a1afb6c655d060d\": container with ID starting with 2497a9ee628af8c7926f5fc4916b0f146e1ce0860fdcb6da0a1afb6c655d060d not found: ID does not exist"
Sep 17 23:43:57.502941 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.502899    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" path="/var/lib/kubelet/pods/4230922d-d05b-40ad-9f23-a5dcc584ae53/volumes"
Sep 17 23:43:57.655320 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.654378    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"]
Sep 17 23:43:57.655320 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.654674    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container" containerID="cri-o://22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf" gracePeriod=30
Sep 17 23:43:57.655320 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.655021    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kube-rbac-proxy" containerID="cri-o://d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f" gracePeriod=30
Sep 17 23:43:57.691650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.689915    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"]
Sep 17 23:43:57.691650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.690317    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="storage-initializer"
Sep 17 23:43:57.691650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.690334    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="storage-initializer"
Sep 17 23:43:57.691650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.690352    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container"
Sep 17 23:43:57.691650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.690362    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container"
Sep 17 23:43:57.691650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.690372    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kube-rbac-proxy"
Sep 17 23:43:57.691650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.690381    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kube-rbac-proxy"
Sep 17 23:43:57.691650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.690464    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kserve-container"
Sep 17 23:43:57.691650 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.690474    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4230922d-d05b-40ad-9f23-a5dcc584ae53" containerName="kube-rbac-proxy"
Sep 17 23:43:57.694140 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.694118    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.698490 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.698469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 23:43:57.704658 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.704535    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"]
Sep 17 23:43:57.801829 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.801518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f506825-8b06-4842-911b-903aa924da1c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.801829 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.801574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f506825-8b06-4842-911b-903aa924da1c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.801829 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.801686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f506825-8b06-4842-911b-903aa924da1c-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.801829 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.801718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f506825-8b06-4842-911b-903aa924da1c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.801829 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.801761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9q8j\" (UniqueName: \"kubernetes.io/projected/7f506825-8b06-4842-911b-903aa924da1c-kube-api-access-w9q8j\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.902976 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.902941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f506825-8b06-4842-911b-903aa924da1c-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.903165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.902997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f506825-8b06-4842-911b-903aa924da1c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.903165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.903040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9q8j\" (UniqueName: \"kubernetes.io/projected/7f506825-8b06-4842-911b-903aa924da1c-kube-api-access-w9q8j\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.903165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.903095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f506825-8b06-4842-911b-903aa924da1c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.903165 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.903126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f506825-8b06-4842-911b-903aa924da1c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.906391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.903890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f506825-8b06-4842-911b-903aa924da1c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.906391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.905225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f506825-8b06-4842-911b-903aa924da1c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.906391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.906131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f506825-8b06-4842-911b-903aa924da1c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.922800 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.922741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9q8j\" (UniqueName: \"kubernetes.io/projected/7f506825-8b06-4842-911b-903aa924da1c-kube-api-access-w9q8j\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.926354 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.926138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f506825-8b06-4842-911b-903aa924da1c-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-xllbf\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:43:57.958020 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.957727    2580 generic.go:356] "Generic (PLEG): container finished" podID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerID="d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f" exitCode=2
Sep 17 23:43:57.958020 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:57.957892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" event={"ID":"2bf64c5b-051c-482c-9a27-cd322ea6ae01","Type":"ContainerDied","Data":"d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f"}
Sep 17 23:43:58.010813 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:43:58.010429    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:44:00.636796 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:00.636260    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 23:44:00.640926 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:00.640863    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:44:01.489319 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:01.488779    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"]
Sep 17 23:44:04.315858 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:44:04.313559    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f506825_8b06_4842_911b_903aa924da1c.slice/crio-d58e2e9157d4e542ddd128b65f1f12df8c4d28d31c1aaa54287b36c61b865bdd WatchSource:0}: Error finding container d58e2e9157d4e542ddd128b65f1f12df8c4d28d31c1aaa54287b36c61b865bdd: Status 404 returned error can't find the container with id d58e2e9157d4e542ddd128b65f1f12df8c4d28d31c1aaa54287b36c61b865bdd
Sep 17 23:44:04.526379 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.526351    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:44:04.665584 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.665439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bf64c5b-051c-482c-9a27-cd322ea6ae01-cabundle-cert\") pod \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") "
Sep 17 23:44:04.665584 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.665543    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kserve-provision-location\") pod \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") "
Sep 17 23:44:04.665584 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.665577    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqspm\" (UniqueName: \"kubernetes.io/projected/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kube-api-access-qqspm\") pod \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") "
Sep 17 23:44:04.665912 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.665602    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kube-rbac-proxy-sar-config\") pod \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") "
Sep 17 23:44:04.665912 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.665695    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bf64c5b-051c-482c-9a27-cd322ea6ae01-proxy-tls\") pod \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\" (UID: \"2bf64c5b-051c-482c-9a27-cd322ea6ae01\") "
Sep 17 23:44:04.667059 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.667029    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf64c5b-051c-482c-9a27-cd322ea6ae01-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2bf64c5b-051c-482c-9a27-cd322ea6ae01" (UID: "2bf64c5b-051c-482c-9a27-cd322ea6ae01"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:44:04.667178 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.667139    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2bf64c5b-051c-482c-9a27-cd322ea6ae01" (UID: "2bf64c5b-051c-482c-9a27-cd322ea6ae01"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:44:04.667178 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.667168    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2bf64c5b-051c-482c-9a27-cd322ea6ae01" (UID: "2bf64c5b-051c-482c-9a27-cd322ea6ae01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:04.675522 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.675491    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf64c5b-051c-482c-9a27-cd322ea6ae01-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2bf64c5b-051c-482c-9a27-cd322ea6ae01" (UID: "2bf64c5b-051c-482c-9a27-cd322ea6ae01"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:44:04.679513 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.679489    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kube-api-access-qqspm" (OuterVolumeSpecName: "kube-api-access-qqspm") pod "2bf64c5b-051c-482c-9a27-cd322ea6ae01" (UID: "2bf64c5b-051c-482c-9a27-cd322ea6ae01"). InnerVolumeSpecName "kube-api-access-qqspm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:44:04.766755 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.766716    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bf64c5b-051c-482c-9a27-cd322ea6ae01-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:44:04.766755 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.766758    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:44:04.767010 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.766775    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqspm\" (UniqueName: \"kubernetes.io/projected/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kube-api-access-qqspm\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:44:04.767010 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.766792    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bf64c5b-051c-482c-9a27-cd322ea6ae01-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:44:04.767010 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:04.766806    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bf64c5b-051c-482c-9a27-cd322ea6ae01-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:44:05.011152 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.007742    2580 generic.go:356] "Generic (PLEG): container finished" podID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerID="22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf" exitCode=0
Sep 17 23:44:05.011152 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.007876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" event={"ID":"2bf64c5b-051c-482c-9a27-cd322ea6ae01","Type":"ContainerDied","Data":"22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf"}
Sep 17 23:44:05.011152 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.007935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr" event={"ID":"2bf64c5b-051c-482c-9a27-cd322ea6ae01","Type":"ContainerDied","Data":"8beff92f99e2c284246fe60179cafb3814c92a3679e6419c487abf293574d1d3"}
Sep 17 23:44:05.011152 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.007977    2580 scope.go:117] "RemoveContainer" containerID="d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f"
Sep 17 23:44:05.011152 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.008128    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"
Sep 17 23:44:05.017389 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.017363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" event={"ID":"7f506825-8b06-4842-911b-903aa924da1c","Type":"ContainerStarted","Data":"eacc2e99102c0874f1976248e56a604391502ce06b1cb351dd84a4fe10ef77bc"}
Sep 17 23:44:05.018232 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.018214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" event={"ID":"7f506825-8b06-4842-911b-903aa924da1c","Type":"ContainerStarted","Data":"d58e2e9157d4e542ddd128b65f1f12df8c4d28d31c1aaa54287b36c61b865bdd"}
Sep 17 23:44:05.030900 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.030874    2580 scope.go:117] "RemoveContainer" containerID="22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf"
Sep 17 23:44:05.053382 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.053203    2580 scope.go:117] "RemoveContainer" containerID="261d5da83bdef676afe368f76ecc432c83861327a688609f42a4fe24696b2282"
Sep 17 23:44:05.059891 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.059868    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"]
Sep 17 23:44:05.060056 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.060043    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-7gbpr"]
Sep 17 23:44:05.075966 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.075400    2580 scope.go:117] "RemoveContainer" containerID="d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f"
Sep 17 23:44:05.075966 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:44:05.075823    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f\": container with ID starting with d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f not found: ID does not exist" containerID="d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f"
Sep 17 23:44:05.075966 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.075859    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f"} err="failed to get container status \"d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f\": rpc error: code = NotFound desc = could not find container \"d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f\": container with ID starting with d1e1d17e6f43493aff14e54766a8ff0139c83d095472bb37fbc8d0dae3d3f80f not found: ID does not exist"
Sep 17 23:44:05.075966 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.075886    2580 scope.go:117] "RemoveContainer" containerID="22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf"
Sep 17 23:44:05.076566 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:44:05.076444    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf\": container with ID starting with 22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf not found: ID does not exist" containerID="22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf"
Sep 17 23:44:05.076566 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.076474    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf"} err="failed to get container status \"22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf\": rpc error: code = NotFound desc = could not find container \"22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf\": container with ID starting with 22675b1153dd6a88b9e040f2c76d82a628fc8188743bd399988cf1a1f59c39bf not found: ID does not exist"
Sep 17 23:44:05.076566 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.076494    2580 scope.go:117] "RemoveContainer" containerID="261d5da83bdef676afe368f76ecc432c83861327a688609f42a4fe24696b2282"
Sep 17 23:44:05.077278 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:44:05.077211    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261d5da83bdef676afe368f76ecc432c83861327a688609f42a4fe24696b2282\": container with ID starting with 261d5da83bdef676afe368f76ecc432c83861327a688609f42a4fe24696b2282 not found: ID does not exist" containerID="261d5da83bdef676afe368f76ecc432c83861327a688609f42a4fe24696b2282"
Sep 17 23:44:05.077278 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.077241    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261d5da83bdef676afe368f76ecc432c83861327a688609f42a4fe24696b2282"} err="failed to get container status \"261d5da83bdef676afe368f76ecc432c83861327a688609f42a4fe24696b2282\": rpc error: code = NotFound desc = could not find container \"261d5da83bdef676afe368f76ecc432c83861327a688609f42a4fe24696b2282\": container with ID starting with 261d5da83bdef676afe368f76ecc432c83861327a688609f42a4fe24696b2282 not found: ID does not exist"
Sep 17 23:44:05.512465 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:05.511990    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" path="/var/lib/kubelet/pods/2bf64c5b-051c-482c-9a27-cd322ea6ae01/volumes"
Sep 17 23:44:09.040549 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:09.040512    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f506825-8b06-4842-911b-903aa924da1c" containerID="eacc2e99102c0874f1976248e56a604391502ce06b1cb351dd84a4fe10ef77bc" exitCode=0
Sep 17 23:44:09.041084 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:09.040634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" event={"ID":"7f506825-8b06-4842-911b-903aa924da1c","Type":"ContainerDied","Data":"eacc2e99102c0874f1976248e56a604391502ce06b1cb351dd84a4fe10ef77bc"}
Sep 17 23:44:10.049103 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:10.049065    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" event={"ID":"7f506825-8b06-4842-911b-903aa924da1c","Type":"ContainerStarted","Data":"a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148"}
Sep 17 23:44:10.049563 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:10.049138    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" event={"ID":"7f506825-8b06-4842-911b-903aa924da1c","Type":"ContainerStarted","Data":"23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f"}
Sep 17 23:44:10.049671 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:10.049569    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:44:10.049671 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:10.049663    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:44:10.051539 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:10.051291    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:44:10.069493 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:10.069435    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podStartSLOduration=8.825325515 podStartE2EDuration="13.069413869s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.244088354s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:44:10.067339007 +0000 UTC m=+1983.111715174" watchObservedRunningTime="2026-09-17 23:44:10.069413869 +0000 UTC m=+1983.113790036"
Sep 17 23:44:11.055519 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:11.055478    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:44:16.061214 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:16.061177    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:44:16.062210 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:16.062179    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:44:26.062300 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:26.062051    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:44:36.062428 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:36.062212    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:44:46.062171 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:46.062125    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:44:56.062400 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:44:56.062354    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:45:06.062341 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:06.062300    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:45:16.061862 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:16.061807    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:45:26.062797 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:26.062748    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:45:35.497139 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:35.497050    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:45:45.500628 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:45.500589    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:45:47.925734 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:47.925705    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"]
Sep 17 23:45:47.926098 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:47.925920    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container" containerID="cri-o://23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f" gracePeriod=30
Sep 17 23:45:47.926098 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:47.925975    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kube-rbac-proxy" containerID="cri-o://a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148" gracePeriod=30
Sep 17 23:45:48.460177 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:48.460105    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f506825-8b06-4842-911b-903aa924da1c" containerID="a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148" exitCode=2
Sep 17 23:45:48.460376 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:48.460175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" event={"ID":"7f506825-8b06-4842-911b-903aa924da1c","Type":"ContainerDied","Data":"a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148"}
Sep 17 23:45:48.462283 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:48.462261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" event={"ID":"40368dd0-36f7-4e45-8aa3-20103cca9dfd","Type":"ContainerStarted","Data":"f5601cf3c4ab80f2430c1d88e353bb57be03135a82635062a3c2f179a0797ab9"}
Sep 17 23:45:48.462424 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:48.462304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" event={"ID":"40368dd0-36f7-4e45-8aa3-20103cca9dfd","Type":"ContainerStarted","Data":"73d4743964f05372dab0de0c72cf3d2bc50fe1b5ea02aed883d9d2239aff964d"}
Sep 17 23:45:48.462424 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:48.462388    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:45:48.482526 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:48.482480    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" podStartSLOduration=2.799574566 podStartE2EDuration="2m1.482468591s" totalImagesPullingTime="1m54.900521767s" totalInitContainerRuntime="3.782372258s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:45:48.480537535 +0000 UTC m=+2081.524913706" watchObservedRunningTime="2026-09-17 23:45:48.482468591 +0000 UTC m=+2081.526844750"
Sep 17 23:45:49.466160 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:49.466124    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:45:49.467154 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:49.467124    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 23:45:50.469918 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:50.469872    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 23:45:51.056120 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:51.056077    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 23:45:52.717442 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.717409    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:45:52.804552 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.804516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f506825-8b06-4842-911b-903aa924da1c-proxy-tls\") pod \"7f506825-8b06-4842-911b-903aa924da1c\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") "
Sep 17 23:45:52.804552 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.804558    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f506825-8b06-4842-911b-903aa924da1c-cabundle-cert\") pod \"7f506825-8b06-4842-911b-903aa924da1c\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") "
Sep 17 23:45:52.804891 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.804576    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9q8j\" (UniqueName: \"kubernetes.io/projected/7f506825-8b06-4842-911b-903aa924da1c-kube-api-access-w9q8j\") pod \"7f506825-8b06-4842-911b-903aa924da1c\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") "
Sep 17 23:45:52.804891 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.804701    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f506825-8b06-4842-911b-903aa924da1c-kube-rbac-proxy-sar-config\") pod \"7f506825-8b06-4842-911b-903aa924da1c\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") "
Sep 17 23:45:52.804891 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.804750    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f506825-8b06-4842-911b-903aa924da1c-kserve-provision-location\") pod \"7f506825-8b06-4842-911b-903aa924da1c\" (UID: \"7f506825-8b06-4842-911b-903aa924da1c\") "
Sep 17 23:45:52.805030 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.804928    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f506825-8b06-4842-911b-903aa924da1c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f506825-8b06-4842-911b-903aa924da1c" (UID: "7f506825-8b06-4842-911b-903aa924da1c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:45:52.805096 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.805079    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f506825-8b06-4842-911b-903aa924da1c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f506825-8b06-4842-911b-903aa924da1c" (UID: "7f506825-8b06-4842-911b-903aa924da1c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:45:52.805202 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.805180    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f506825-8b06-4842-911b-903aa924da1c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f506825-8b06-4842-911b-903aa924da1c" (UID: "7f506825-8b06-4842-911b-903aa924da1c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:52.806856 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.806827    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f506825-8b06-4842-911b-903aa924da1c-kube-api-access-w9q8j" (OuterVolumeSpecName: "kube-api-access-w9q8j") pod "7f506825-8b06-4842-911b-903aa924da1c" (UID: "7f506825-8b06-4842-911b-903aa924da1c"). InnerVolumeSpecName "kube-api-access-w9q8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:45:52.806964 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.806906    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f506825-8b06-4842-911b-903aa924da1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f506825-8b06-4842-911b-903aa924da1c" (UID: "7f506825-8b06-4842-911b-903aa924da1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:45:52.906011 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.905982    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f506825-8b06-4842-911b-903aa924da1c-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:45:52.906011 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.906008    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9q8j\" (UniqueName: \"kubernetes.io/projected/7f506825-8b06-4842-911b-903aa924da1c-kube-api-access-w9q8j\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:45:52.906233 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.906024    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f506825-8b06-4842-911b-903aa924da1c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:45:52.906233 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.906034    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f506825-8b06-4842-911b-903aa924da1c-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:45:52.906233 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:52.906043    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f506825-8b06-4842-911b-903aa924da1c-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:45:53.035206 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035171    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"]
Sep 17 23:45:53.035515 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035502    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="storage-initializer"
Sep 17 23:45:53.035560 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035517    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="storage-initializer"
Sep 17 23:45:53.035560 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035529    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="storage-initializer"
Sep 17 23:45:53.035560 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035534    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="storage-initializer"
Sep 17 23:45:53.035560 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035542    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container"
Sep 17 23:45:53.035560 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035547    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container"
Sep 17 23:45:53.035560 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035553    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kube-rbac-proxy"
Sep 17 23:45:53.035560 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035557    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kube-rbac-proxy"
Sep 17 23:45:53.035790 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035565    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container"
Sep 17 23:45:53.035790 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035570    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container"
Sep 17 23:45:53.035790 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035576    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kube-rbac-proxy"
Sep 17 23:45:53.035790 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035581    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kube-rbac-proxy"
Sep 17 23:45:53.035790 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035637    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kube-rbac-proxy"
Sep 17 23:45:53.035790 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035649    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f506825-8b06-4842-911b-903aa924da1c" containerName="kserve-container"
Sep 17 23:45:53.035790 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035656    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kube-rbac-proxy"
Sep 17 23:45:53.035790 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.035665    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bf64c5b-051c-482c-9a27-cd322ea6ae01" containerName="kserve-container"
Sep 17 23:45:53.059811 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.059746    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"]
Sep 17 23:45:53.059937 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.059864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.062372 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.062353    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 23:45:53.062469 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.062355    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 23:45:53.208799 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.208766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlk7j\" (UniqueName: \"kubernetes.io/projected/4ee31188-b636-474a-b5f0-a880f3ea2101-kube-api-access-hlk7j\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.208969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.208823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ee31188-b636-474a-b5f0-a880f3ea2101-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.208969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.208873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ee31188-b636-474a-b5f0-a880f3ea2101-cabundle-cert\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.208969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.208892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ee31188-b636-474a-b5f0-a880f3ea2101-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.208969 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.208915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ee31188-b636-474a-b5f0-a880f3ea2101-kserve-provision-location\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.309973 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.309887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ee31188-b636-474a-b5f0-a880f3ea2101-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.309973 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.309921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ee31188-b636-474a-b5f0-a880f3ea2101-kserve-provision-location\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.309973 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.309959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlk7j\" (UniqueName: \"kubernetes.io/projected/4ee31188-b636-474a-b5f0-a880f3ea2101-kube-api-access-hlk7j\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.310234 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.309992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ee31188-b636-474a-b5f0-a880f3ea2101-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.310234 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.310016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ee31188-b636-474a-b5f0-a880f3ea2101-cabundle-cert\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.310367 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.310347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ee31188-b636-474a-b5f0-a880f3ea2101-kserve-provision-location\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.310727 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.310703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ee31188-b636-474a-b5f0-a880f3ea2101-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.310795 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.310729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ee31188-b636-474a-b5f0-a880f3ea2101-cabundle-cert\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.312562 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.312536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ee31188-b636-474a-b5f0-a880f3ea2101-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.319215 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.319195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlk7j\" (UniqueName: \"kubernetes.io/projected/4ee31188-b636-474a-b5f0-a880f3ea2101-kube-api-access-hlk7j\") pod \"isvc-canary-stop-predictor-596476db66-wct5g\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.370022 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.369993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:53.483716 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.483686    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f506825-8b06-4842-911b-903aa924da1c" containerID="23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f" exitCode=0
Sep 17 23:45:53.483834 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.483754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" event={"ID":"7f506825-8b06-4842-911b-903aa924da1c","Type":"ContainerDied","Data":"23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f"}
Sep 17 23:45:53.483881 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.483830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf" event={"ID":"7f506825-8b06-4842-911b-903aa924da1c","Type":"ContainerDied","Data":"d58e2e9157d4e542ddd128b65f1f12df8c4d28d31c1aaa54287b36c61b865bdd"}
Sep 17 23:45:53.483881 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.483765    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"
Sep 17 23:45:53.483881 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.483870    2580 scope.go:117] "RemoveContainer" containerID="a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148"
Sep 17 23:45:53.504351 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.504332    2580 scope.go:117] "RemoveContainer" containerID="23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f"
Sep 17 23:45:53.511575 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.511554    2580 scope.go:117] "RemoveContainer" containerID="eacc2e99102c0874f1976248e56a604391502ce06b1cb351dd84a4fe10ef77bc"
Sep 17 23:45:53.516328 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.516306    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"]
Sep 17 23:45:53.520305 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.520286    2580 scope.go:117] "RemoveContainer" containerID="a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148"
Sep 17 23:45:53.520508 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.520491    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-xllbf"]
Sep 17 23:45:53.520604 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:45:53.520588    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148\": container with ID starting with a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148 not found: ID does not exist" containerID="a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148"
Sep 17 23:45:53.520705 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.520641    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148"} err="failed to get container status \"a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148\": rpc error: code = NotFound desc = could not find container \"a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148\": container with ID starting with a2edafe75b275ac043f7707477c476d5d65d4178ac72b458513711503da7c148 not found: ID does not exist"
Sep 17 23:45:53.520705 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.520668    2580 scope.go:117] "RemoveContainer" containerID="23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f"
Sep 17 23:45:53.520919 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:45:53.520901    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f\": container with ID starting with 23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f not found: ID does not exist" containerID="23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f"
Sep 17 23:45:53.520961 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.520926    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f"} err="failed to get container status \"23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f\": rpc error: code = NotFound desc = could not find container \"23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f\": container with ID starting with 23bb7a0aa04e57ee1c1576934c8dcb510ac8844c7018958b1d9418294c6ae31f not found: ID does not exist"
Sep 17 23:45:53.520961 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.520942    2580 scope.go:117] "RemoveContainer" containerID="eacc2e99102c0874f1976248e56a604391502ce06b1cb351dd84a4fe10ef77bc"
Sep 17 23:45:53.521190 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:45:53.521177    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eacc2e99102c0874f1976248e56a604391502ce06b1cb351dd84a4fe10ef77bc\": container with ID starting with eacc2e99102c0874f1976248e56a604391502ce06b1cb351dd84a4fe10ef77bc not found: ID does not exist" containerID="eacc2e99102c0874f1976248e56a604391502ce06b1cb351dd84a4fe10ef77bc"
Sep 17 23:45:53.521233 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.521193    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eacc2e99102c0874f1976248e56a604391502ce06b1cb351dd84a4fe10ef77bc"} err="failed to get container status \"eacc2e99102c0874f1976248e56a604391502ce06b1cb351dd84a4fe10ef77bc\": rpc error: code = NotFound desc = could not find container \"eacc2e99102c0874f1976248e56a604391502ce06b1cb351dd84a4fe10ef77bc\": container with ID starting with eacc2e99102c0874f1976248e56a604391502ce06b1cb351dd84a4fe10ef77bc not found: ID does not exist"
Sep 17 23:45:53.699550 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:53.699497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"]
Sep 17 23:45:53.701790 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:45:53.701759    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee31188_b636_474a_b5f0_a880f3ea2101.slice/crio-6065bb0a3a26bd1770ee98d539b920cb9b590200d9f7a57bd55f5d60d20c51a7 WatchSource:0}: Error finding container 6065bb0a3a26bd1770ee98d539b920cb9b590200d9f7a57bd55f5d60d20c51a7: Status 404 returned error can't find the container with id 6065bb0a3a26bd1770ee98d539b920cb9b590200d9f7a57bd55f5d60d20c51a7
Sep 17 23:45:54.489082 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:54.489047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" event={"ID":"4ee31188-b636-474a-b5f0-a880f3ea2101","Type":"ContainerStarted","Data":"443fcfec6a57387fb7bf345201994c3a973c3095e96d2339f4400cc880682c50"}
Sep 17 23:45:54.489533 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:54.489118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" event={"ID":"4ee31188-b636-474a-b5f0-a880f3ea2101","Type":"ContainerStarted","Data":"6065bb0a3a26bd1770ee98d539b920cb9b590200d9f7a57bd55f5d60d20c51a7"}
Sep 17 23:45:55.476250 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:55.476222    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:45:55.476948 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:55.476930    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:45:55.499904 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:55.499881    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f506825-8b06-4842-911b-903aa924da1c" path="/var/lib/kubelet/pods/7f506825-8b06-4842-911b-903aa924da1c/volumes"
Sep 17 23:45:58.506278 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:58.506246    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerID="443fcfec6a57387fb7bf345201994c3a973c3095e96d2339f4400cc880682c50" exitCode=0
Sep 17 23:45:58.506754 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:58.506322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" event={"ID":"4ee31188-b636-474a-b5f0-a880f3ea2101","Type":"ContainerDied","Data":"443fcfec6a57387fb7bf345201994c3a973c3095e96d2339f4400cc880682c50"}
Sep 17 23:45:59.511442 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:59.511410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" event={"ID":"4ee31188-b636-474a-b5f0-a880f3ea2101","Type":"ContainerStarted","Data":"242021d309f43cbe8aab3a381dc1c84a8d8a2783ce78c513beff271bb4b3048a"}
Sep 17 23:45:59.511956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:59.511467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" event={"ID":"4ee31188-b636-474a-b5f0-a880f3ea2101","Type":"ContainerStarted","Data":"f6660d7992082a70062bb9c515d3052935e9ac1fc296824c5ae6f7369ebce933"}
Sep 17 23:45:59.511956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:59.511725    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:59.511956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:59.511869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:45:59.513080 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:59.513051    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 23:45:59.533048 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:59.533002    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podStartSLOduration=2.030511919 podStartE2EDuration="6.532989439s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.50247752s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:45:59.531628472 +0000 UTC m=+2092.576004625" watchObservedRunningTime="2026-09-17 23:45:59.532989439 +0000 UTC m=+2092.577365596"
Sep 17 23:45:59.870713 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:59.870679    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"]
Sep 17 23:45:59.870991 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:59.870962    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="kserve-container" containerID="cri-o://73d4743964f05372dab0de0c72cf3d2bc50fe1b5ea02aed883d9d2239aff964d" gracePeriod=30
Sep 17 23:45:59.871465 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:45:59.870985    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="kube-rbac-proxy" containerID="cri-o://f5601cf3c4ab80f2430c1d88e353bb57be03135a82635062a3c2f179a0797ab9" gracePeriod=30
Sep 17 23:46:00.470757 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:00.470705    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 17 23:46:00.516525 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:00.516492    2580 generic.go:356] "Generic (PLEG): container finished" podID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerID="f5601cf3c4ab80f2430c1d88e353bb57be03135a82635062a3c2f179a0797ab9" exitCode=2
Sep 17 23:46:00.516910 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:00.516563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" event={"ID":"40368dd0-36f7-4e45-8aa3-20103cca9dfd","Type":"ContainerDied","Data":"f5601cf3c4ab80f2430c1d88e353bb57be03135a82635062a3c2f179a0797ab9"}
Sep 17 23:46:00.516960 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:00.516922    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 23:46:02.526066 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:02.526029    2580 generic.go:356] "Generic (PLEG): container finished" podID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerID="73d4743964f05372dab0de0c72cf3d2bc50fe1b5ea02aed883d9d2239aff964d" exitCode=0
Sep 17 23:46:02.526434 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:02.526072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" event={"ID":"40368dd0-36f7-4e45-8aa3-20103cca9dfd","Type":"ContainerDied","Data":"73d4743964f05372dab0de0c72cf3d2bc50fe1b5ea02aed883d9d2239aff964d"}
Sep 17 23:46:03.039240 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.039207    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:46:03.190277 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.190199    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kserve-provision-location\") pod \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") "
Sep 17 23:46:03.190277 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.190254    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txr2s\" (UniqueName: \"kubernetes.io/projected/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kube-api-access-txr2s\") pod \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") "
Sep 17 23:46:03.190277 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.190279    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kube-rbac-proxy-sar-config\") pod \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") "
Sep 17 23:46:03.190533 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.190317    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40368dd0-36f7-4e45-8aa3-20103cca9dfd-proxy-tls\") pod \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") "
Sep 17 23:46:03.190533 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.190346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40368dd0-36f7-4e45-8aa3-20103cca9dfd-cabundle-cert\") pod \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\" (UID: \"40368dd0-36f7-4e45-8aa3-20103cca9dfd\") "
Sep 17 23:46:03.190744 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.190717    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40368dd0-36f7-4e45-8aa3-20103cca9dfd" (UID: "40368dd0-36f7-4e45-8aa3-20103cca9dfd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:46:03.190857 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.190709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40368dd0-36f7-4e45-8aa3-20103cca9dfd" (UID: "40368dd0-36f7-4e45-8aa3-20103cca9dfd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:46:03.190857 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.190812    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40368dd0-36f7-4e45-8aa3-20103cca9dfd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40368dd0-36f7-4e45-8aa3-20103cca9dfd" (UID: "40368dd0-36f7-4e45-8aa3-20103cca9dfd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:46:03.192488 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.192467    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40368dd0-36f7-4e45-8aa3-20103cca9dfd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40368dd0-36f7-4e45-8aa3-20103cca9dfd" (UID: "40368dd0-36f7-4e45-8aa3-20103cca9dfd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:46:03.192559 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.192489    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kube-api-access-txr2s" (OuterVolumeSpecName: "kube-api-access-txr2s") pod "40368dd0-36f7-4e45-8aa3-20103cca9dfd" (UID: "40368dd0-36f7-4e45-8aa3-20103cca9dfd"). InnerVolumeSpecName "kube-api-access-txr2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:46:03.291070 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.291040    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txr2s\" (UniqueName: \"kubernetes.io/projected/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kube-api-access-txr2s\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:46:03.291070 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.291065    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:46:03.291070 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.291074    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40368dd0-36f7-4e45-8aa3-20103cca9dfd-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:46:03.291271 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.291083    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40368dd0-36f7-4e45-8aa3-20103cca9dfd-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:46:03.291271 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.291101    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40368dd0-36f7-4e45-8aa3-20103cca9dfd-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:46:03.530940 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.530906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj" event={"ID":"40368dd0-36f7-4e45-8aa3-20103cca9dfd","Type":"ContainerDied","Data":"b48bab6b0e2471e54f3c2c7e3d2260750b35745e1ae4f4bfd16790e5a1253b28"}
Sep 17 23:46:03.531294 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.530967    2580 scope.go:117] "RemoveContainer" containerID="f5601cf3c4ab80f2430c1d88e353bb57be03135a82635062a3c2f179a0797ab9"
Sep 17 23:46:03.531294 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.530918    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"
Sep 17 23:46:03.538686 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.538661    2580 scope.go:117] "RemoveContainer" containerID="73d4743964f05372dab0de0c72cf3d2bc50fe1b5ea02aed883d9d2239aff964d"
Sep 17 23:46:03.545351 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.545334    2580 scope.go:117] "RemoveContainer" containerID="a9b1a91201cd9ef6ce39e25e5cc59d5dc26a7678cc3ca0df8fc32a669103fd82"
Sep 17 23:46:03.549744 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.549719    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"]
Sep 17 23:46:03.555668 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:03.555648    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-frcfj"]
Sep 17 23:46:05.048275 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.048241    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"]
Sep 17 23:46:05.048653 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.048496    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="kserve-container"
Sep 17 23:46:05.048653 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.048507    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="kserve-container"
Sep 17 23:46:05.048653 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.048516    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="storage-initializer"
Sep 17 23:46:05.048653 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.048523    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="storage-initializer"
Sep 17 23:46:05.048653 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.048529    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="kube-rbac-proxy"
Sep 17 23:46:05.048653 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.048535    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="kube-rbac-proxy"
Sep 17 23:46:05.048653 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.048578    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="kserve-container"
Sep 17 23:46:05.048653 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.048585    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" containerName="kube-rbac-proxy"
Sep 17 23:46:05.051817 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.051799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.054139 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.054121    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 17 23:46:05.054139 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.054129    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 23:46:05.061379 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.061350    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"]
Sep 17 23:46:05.205300 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.205260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3abb2655-44b0-4347-86e4-3f4f308b896d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.205476 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.205333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3abb2655-44b0-4347-86e4-3f4f308b896d-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.205476 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.205393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3abb2655-44b0-4347-86e4-3f4f308b896d-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.205476 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.205439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3abb2655-44b0-4347-86e4-3f4f308b896d-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.205658 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.205486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdgx\" (UniqueName: \"kubernetes.io/projected/3abb2655-44b0-4347-86e4-3f4f308b896d-kube-api-access-czdgx\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.306170 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.306079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3abb2655-44b0-4347-86e4-3f4f308b896d-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.306170 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.306131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdgx\" (UniqueName: \"kubernetes.io/projected/3abb2655-44b0-4347-86e4-3f4f308b896d-kube-api-access-czdgx\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.306170 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.306168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3abb2655-44b0-4347-86e4-3f4f308b896d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.306441 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.306309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3abb2655-44b0-4347-86e4-3f4f308b896d-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.306441 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.306347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3abb2655-44b0-4347-86e4-3f4f308b896d-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.306673 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.306656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3abb2655-44b0-4347-86e4-3f4f308b896d-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.306868 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.306849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3abb2655-44b0-4347-86e4-3f4f308b896d-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.306989 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.306970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3abb2655-44b0-4347-86e4-3f4f308b896d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.308902 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.308881    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3abb2655-44b0-4347-86e4-3f4f308b896d-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.314885 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.314867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdgx\" (UniqueName: \"kubernetes.io/projected/3abb2655-44b0-4347-86e4-3f4f308b896d-kube-api-access-czdgx\") pod \"isvc-xgboost-predictor-7566d65bf7-5rdjs\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.361895 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.361857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:05.500460 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.500430    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40368dd0-36f7-4e45-8aa3-20103cca9dfd" path="/var/lib/kubelet/pods/40368dd0-36f7-4e45-8aa3-20103cca9dfd/volumes"
Sep 17 23:46:05.522241 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.522207    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:46:05.527477 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.523688    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 23:46:05.631751 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:05.631675    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"]
Sep 17 23:46:05.634921 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:46:05.634894    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abb2655_44b0_4347_86e4_3f4f308b896d.slice/crio-999dc869ba0b981467f277aa0cd84bbcb4378acea908ad2b3fa0162f07c843d2 WatchSource:0}: Error finding container 999dc869ba0b981467f277aa0cd84bbcb4378acea908ad2b3fa0162f07c843d2: Status 404 returned error can't find the container with id 999dc869ba0b981467f277aa0cd84bbcb4378acea908ad2b3fa0162f07c843d2
Sep 17 23:46:06.543856 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:06.543822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" event={"ID":"3abb2655-44b0-4347-86e4-3f4f308b896d","Type":"ContainerStarted","Data":"bfa22e41d965e76419d2137c55151fffa73a1c782220a2b32992e41aded22a23"}
Sep 17 23:46:06.544318 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:06.543878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" event={"ID":"3abb2655-44b0-4347-86e4-3f4f308b896d","Type":"ContainerStarted","Data":"999dc869ba0b981467f277aa0cd84bbcb4378acea908ad2b3fa0162f07c843d2"}
Sep 17 23:46:10.559718 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:10.559680    2580 generic.go:356] "Generic (PLEG): container finished" podID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerID="bfa22e41d965e76419d2137c55151fffa73a1c782220a2b32992e41aded22a23" exitCode=0
Sep 17 23:46:10.560092 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:10.559754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" event={"ID":"3abb2655-44b0-4347-86e4-3f4f308b896d","Type":"ContainerDied","Data":"bfa22e41d965e76419d2137c55151fffa73a1c782220a2b32992e41aded22a23"}
Sep 17 23:46:15.523226 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:15.523185    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 23:46:25.523296 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:25.523240    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 23:46:29.636760 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:29.636728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" event={"ID":"3abb2655-44b0-4347-86e4-3f4f308b896d","Type":"ContainerStarted","Data":"f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432"}
Sep 17 23:46:29.637169 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:29.636798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" event={"ID":"3abb2655-44b0-4347-86e4-3f4f308b896d","Type":"ContainerStarted","Data":"db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856"}
Sep 17 23:46:29.637169 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:29.636993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:29.659215 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:29.659167    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podStartSLOduration=1.923206175 podStartE2EDuration="24.659152068s" totalImagesPullingTime="18.6225523s" totalInitContainerRuntime="4.113393593s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:46:29.657578758 +0000 UTC m=+2122.701954939" watchObservedRunningTime="2026-09-17 23:46:29.659152068 +0000 UTC m=+2122.703528226"
Sep 17 23:46:30.640887 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:30.640860    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:30.642180 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:30.642151    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:46:31.644319 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:31.644279    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:46:35.523532 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:35.523495    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 23:46:36.649577 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:36.649547    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:46:36.650227 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:36.650200    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:46:45.523783 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:45.523742    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 23:46:46.650563 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:46.650531    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:46:55.523374 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:55.523333    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 23:46:56.650661 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:46:56.650595    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:47:05.522983 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:05.522888    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 23:47:06.261499 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:06.261466    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"]
Sep 17 23:47:06.261789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:06.261734    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container" containerID="cri-o://f6660d7992082a70062bb9c515d3052935e9ac1fc296824c5ae6f7369ebce933" gracePeriod=30
Sep 17 23:47:06.261932 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:06.261848    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kube-rbac-proxy" containerID="cri-o://242021d309f43cbe8aab3a381dc1c84a8d8a2783ce78c513beff271bb4b3048a" gracePeriod=30
Sep 17 23:47:06.262003 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:06.261975    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 23:47:06.650169 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:06.650134    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:47:06.771023 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:06.770992    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerID="242021d309f43cbe8aab3a381dc1c84a8d8a2783ce78c513beff271bb4b3048a" exitCode=2
Sep 17 23:47:06.771214 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:06.771069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" event={"ID":"4ee31188-b636-474a-b5f0-a880f3ea2101","Type":"ContainerDied","Data":"242021d309f43cbe8aab3a381dc1c84a8d8a2783ce78c513beff271bb4b3048a"}
Sep 17 23:47:10.517498 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:10.517455    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 23:47:15.518007 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:15.517961    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 23:47:16.262850 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:16.262811    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:47:16.650130 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:16.650094    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:47:20.517666 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:20.517604    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 23:47:20.518030 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:20.517747    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:47:25.517793 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:25.517755    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 23:47:26.650548 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:26.650512    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:47:30.517994 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:30.517954    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 23:47:35.517278 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:35.517240    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 23:47:36.651207 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:36.651181    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:47:36.880428 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:36.880400    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerID="f6660d7992082a70062bb9c515d3052935e9ac1fc296824c5ae6f7369ebce933" exitCode=137
Sep 17 23:47:36.880586 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:36.880436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" event={"ID":"4ee31188-b636-474a-b5f0-a880f3ea2101","Type":"ContainerDied","Data":"f6660d7992082a70062bb9c515d3052935e9ac1fc296824c5ae6f7369ebce933"}
Sep 17 23:47:36.902711 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:36.902662    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:47:37.019364 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.019332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlk7j\" (UniqueName: \"kubernetes.io/projected/4ee31188-b636-474a-b5f0-a880f3ea2101-kube-api-access-hlk7j\") pod \"4ee31188-b636-474a-b5f0-a880f3ea2101\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") "
Sep 17 23:47:37.019526 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.019372    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ee31188-b636-474a-b5f0-a880f3ea2101-cabundle-cert\") pod \"4ee31188-b636-474a-b5f0-a880f3ea2101\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") "
Sep 17 23:47:37.019526 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.019401    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ee31188-b636-474a-b5f0-a880f3ea2101-kube-rbac-proxy-sar-config\") pod \"4ee31188-b636-474a-b5f0-a880f3ea2101\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") "
Sep 17 23:47:37.019526 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.019424    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ee31188-b636-474a-b5f0-a880f3ea2101-kserve-provision-location\") pod \"4ee31188-b636-474a-b5f0-a880f3ea2101\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") "
Sep 17 23:47:37.019526 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.019459    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ee31188-b636-474a-b5f0-a880f3ea2101-proxy-tls\") pod \"4ee31188-b636-474a-b5f0-a880f3ea2101\" (UID: \"4ee31188-b636-474a-b5f0-a880f3ea2101\") "
Sep 17 23:47:37.019782 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.019759    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee31188-b636-474a-b5f0-a880f3ea2101-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ee31188-b636-474a-b5f0-a880f3ea2101" (UID: "4ee31188-b636-474a-b5f0-a880f3ea2101"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:47:37.019843 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.019804    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee31188-b636-474a-b5f0-a880f3ea2101-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ee31188-b636-474a-b5f0-a880f3ea2101" (UID: "4ee31188-b636-474a-b5f0-a880f3ea2101"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:47:37.019843 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.019832    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee31188-b636-474a-b5f0-a880f3ea2101-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ee31188-b636-474a-b5f0-a880f3ea2101" (UID: "4ee31188-b636-474a-b5f0-a880f3ea2101"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:47:37.021582 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.021559    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee31188-b636-474a-b5f0-a880f3ea2101-kube-api-access-hlk7j" (OuterVolumeSpecName: "kube-api-access-hlk7j") pod "4ee31188-b636-474a-b5f0-a880f3ea2101" (UID: "4ee31188-b636-474a-b5f0-a880f3ea2101"). InnerVolumeSpecName "kube-api-access-hlk7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:47:37.021711 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.021562    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee31188-b636-474a-b5f0-a880f3ea2101-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ee31188-b636-474a-b5f0-a880f3ea2101" (UID: "4ee31188-b636-474a-b5f0-a880f3ea2101"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:47:37.120132 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.120099    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ee31188-b636-474a-b5f0-a880f3ea2101-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:47:37.120132 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.120127    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ee31188-b636-474a-b5f0-a880f3ea2101-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:47:37.120132 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.120136    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ee31188-b636-474a-b5f0-a880f3ea2101-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:47:37.120350 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.120147    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ee31188-b636-474a-b5f0-a880f3ea2101-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:47:37.120350 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.120155    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlk7j\" (UniqueName: \"kubernetes.io/projected/4ee31188-b636-474a-b5f0-a880f3ea2101-kube-api-access-hlk7j\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:47:37.885727 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.885692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g" event={"ID":"4ee31188-b636-474a-b5f0-a880f3ea2101","Type":"ContainerDied","Data":"6065bb0a3a26bd1770ee98d539b920cb9b590200d9f7a57bd55f5d60d20c51a7"}
Sep 17 23:47:37.886124 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.885746    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"
Sep 17 23:47:37.886124 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.885758    2580 scope.go:117] "RemoveContainer" containerID="242021d309f43cbe8aab3a381dc1c84a8d8a2783ce78c513beff271bb4b3048a"
Sep 17 23:47:37.893288 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.893270    2580 scope.go:117] "RemoveContainer" containerID="f6660d7992082a70062bb9c515d3052935e9ac1fc296824c5ae6f7369ebce933"
Sep 17 23:47:37.899839 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.899823    2580 scope.go:117] "RemoveContainer" containerID="443fcfec6a57387fb7bf345201994c3a973c3095e96d2339f4400cc880682c50"
Sep 17 23:47:37.908003 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.907975    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"]
Sep 17 23:47:37.912156 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:37.912135    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-wct5g"]
Sep 17 23:47:39.500186 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:39.500147    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" path="/var/lib/kubelet/pods/4ee31188-b636-474a-b5f0-a880f3ea2101/volumes"
Sep 17 23:47:45.580233 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:45.580198    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"]
Sep 17 23:47:45.580701 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:45.580506    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container" containerID="cri-o://db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856" gracePeriod=30
Sep 17 23:47:45.580701 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:45.580592    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kube-rbac-proxy" containerID="cri-o://f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432" gracePeriod=30
Sep 17 23:47:45.916138 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:45.916050    2580 generic.go:356] "Generic (PLEG): container finished" podID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerID="f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432" exitCode=2
Sep 17 23:47:45.916138 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:45.916100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" event={"ID":"3abb2655-44b0-4347-86e4-3f4f308b896d","Type":"ContainerDied","Data":"f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432"}
Sep 17 23:47:46.645260 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:46.645212    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 17 23:47:46.650665 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:46.650601    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:47:49.611390 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.611361    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:47:49.711054 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.711022    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3abb2655-44b0-4347-86e4-3f4f308b896d-kube-rbac-proxy-sar-config\") pod \"3abb2655-44b0-4347-86e4-3f4f308b896d\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") "
Sep 17 23:47:49.711221 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.711069    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3abb2655-44b0-4347-86e4-3f4f308b896d-cabundle-cert\") pod \"3abb2655-44b0-4347-86e4-3f4f308b896d\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") "
Sep 17 23:47:49.711221 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.711102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czdgx\" (UniqueName: \"kubernetes.io/projected/3abb2655-44b0-4347-86e4-3f4f308b896d-kube-api-access-czdgx\") pod \"3abb2655-44b0-4347-86e4-3f4f308b896d\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") "
Sep 17 23:47:49.711221 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.711142    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3abb2655-44b0-4347-86e4-3f4f308b896d-proxy-tls\") pod \"3abb2655-44b0-4347-86e4-3f4f308b896d\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") "
Sep 17 23:47:49.711221 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.711160    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3abb2655-44b0-4347-86e4-3f4f308b896d-kserve-provision-location\") pod \"3abb2655-44b0-4347-86e4-3f4f308b896d\" (UID: \"3abb2655-44b0-4347-86e4-3f4f308b896d\") "
Sep 17 23:47:49.711441 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.711416    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abb2655-44b0-4347-86e4-3f4f308b896d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3abb2655-44b0-4347-86e4-3f4f308b896d" (UID: "3abb2655-44b0-4347-86e4-3f4f308b896d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:47:49.711602 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.711574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abb2655-44b0-4347-86e4-3f4f308b896d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3abb2655-44b0-4347-86e4-3f4f308b896d" (UID: "3abb2655-44b0-4347-86e4-3f4f308b896d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:47:49.711602 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.711575    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abb2655-44b0-4347-86e4-3f4f308b896d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3abb2655-44b0-4347-86e4-3f4f308b896d" (UID: "3abb2655-44b0-4347-86e4-3f4f308b896d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:47:49.713352 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.713332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3abb2655-44b0-4347-86e4-3f4f308b896d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3abb2655-44b0-4347-86e4-3f4f308b896d" (UID: "3abb2655-44b0-4347-86e4-3f4f308b896d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:47:49.713497 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.713469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abb2655-44b0-4347-86e4-3f4f308b896d-kube-api-access-czdgx" (OuterVolumeSpecName: "kube-api-access-czdgx") pod "3abb2655-44b0-4347-86e4-3f4f308b896d" (UID: "3abb2655-44b0-4347-86e4-3f4f308b896d"). InnerVolumeSpecName "kube-api-access-czdgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:47:49.812212 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.812185    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3abb2655-44b0-4347-86e4-3f4f308b896d-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:47:49.812212 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.812212    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3abb2655-44b0-4347-86e4-3f4f308b896d-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:47:49.812419 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.812224    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3abb2655-44b0-4347-86e4-3f4f308b896d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:47:49.812419 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.812232    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3abb2655-44b0-4347-86e4-3f4f308b896d-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:47:49.812419 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.812242    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czdgx\" (UniqueName: \"kubernetes.io/projected/3abb2655-44b0-4347-86e4-3f4f308b896d-kube-api-access-czdgx\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:47:49.931862 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.931831    2580 generic.go:356] "Generic (PLEG): container finished" podID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerID="db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856" exitCode=0
Sep 17 23:47:49.932040 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.931891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" event={"ID":"3abb2655-44b0-4347-86e4-3f4f308b896d","Type":"ContainerDied","Data":"db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856"}
Sep 17 23:47:49.932040 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.931901    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"
Sep 17 23:47:49.932040 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.931937    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs" event={"ID":"3abb2655-44b0-4347-86e4-3f4f308b896d","Type":"ContainerDied","Data":"999dc869ba0b981467f277aa0cd84bbcb4378acea908ad2b3fa0162f07c843d2"}
Sep 17 23:47:49.932040 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.931970    2580 scope.go:117] "RemoveContainer" containerID="f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432"
Sep 17 23:47:49.939791 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.939774    2580 scope.go:117] "RemoveContainer" containerID="db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856"
Sep 17 23:47:49.949025 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.949005    2580 scope.go:117] "RemoveContainer" containerID="bfa22e41d965e76419d2137c55151fffa73a1c782220a2b32992e41aded22a23"
Sep 17 23:47:49.952709 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.952688    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"]
Sep 17 23:47:49.956912 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.956894    2580 scope.go:117] "RemoveContainer" containerID="f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432"
Sep 17 23:47:49.956976 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.956943    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5rdjs"]
Sep 17 23:47:49.957221 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:47:49.957201    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432\": container with ID starting with f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432 not found: ID does not exist" containerID="f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432"
Sep 17 23:47:49.957265 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.957229    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432"} err="failed to get container status \"f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432\": rpc error: code = NotFound desc = could not find container \"f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432\": container with ID starting with f1effb6749d8a42be8988104bd131a1d2b56a195e495ee0730e908b25b52f432 not found: ID does not exist"
Sep 17 23:47:49.957265 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.957247    2580 scope.go:117] "RemoveContainer" containerID="db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856"
Sep 17 23:47:49.957568 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:47:49.957546    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856\": container with ID starting with db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856 not found: ID does not exist" containerID="db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856"
Sep 17 23:47:49.957641 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.957570    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856"} err="failed to get container status \"db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856\": rpc error: code = NotFound desc = could not find container \"db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856\": container with ID starting with db1a3b16686d1c57336d45bea4111a151c8dede2a74c4144e2ab080e75ccb856 not found: ID does not exist"
Sep 17 23:47:49.957641 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.957585    2580 scope.go:117] "RemoveContainer" containerID="bfa22e41d965e76419d2137c55151fffa73a1c782220a2b32992e41aded22a23"
Sep 17 23:47:49.957958 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:47:49.957940    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa22e41d965e76419d2137c55151fffa73a1c782220a2b32992e41aded22a23\": container with ID starting with bfa22e41d965e76419d2137c55151fffa73a1c782220a2b32992e41aded22a23 not found: ID does not exist" containerID="bfa22e41d965e76419d2137c55151fffa73a1c782220a2b32992e41aded22a23"
Sep 17 23:47:49.958020 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:49.957969    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa22e41d965e76419d2137c55151fffa73a1c782220a2b32992e41aded22a23"} err="failed to get container status \"bfa22e41d965e76419d2137c55151fffa73a1c782220a2b32992e41aded22a23\": rpc error: code = NotFound desc = could not find container \"bfa22e41d965e76419d2137c55151fffa73a1c782220a2b32992e41aded22a23\": container with ID starting with bfa22e41d965e76419d2137c55151fffa73a1c782220a2b32992e41aded22a23 not found: ID does not exist"
Sep 17 23:47:51.500185 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:47:51.500144    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" path="/var/lib/kubelet/pods/3abb2655-44b0-4347-86e4-3f4f308b896d/volumes"
Sep 17 23:49:09.233466 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233432    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"]
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233715    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="storage-initializer"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233727    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="storage-initializer"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233738    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kube-rbac-proxy"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233744    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kube-rbac-proxy"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233750    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="storage-initializer"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233756    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="storage-initializer"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233765    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233770    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233778    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233784    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233797    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kube-rbac-proxy"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233801    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kube-rbac-proxy"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233846    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kserve-container"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233853    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kserve-container"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233858    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3abb2655-44b0-4347-86e4-3f4f308b896d" containerName="kube-rbac-proxy"
Sep 17 23:49:09.233956 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.233865    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ee31188-b636-474a-b5f0-a880f3ea2101" containerName="kube-rbac-proxy"
Sep 17 23:49:09.236254 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.236237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.238660 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.238634    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 17 23:49:09.238781 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.238751    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:49:09.238959 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.238930    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:49:09.238959 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.238948    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:49:09.239118 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.239063    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:49:09.239725 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.239707    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 17 23:49:09.239790 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.239770    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:49:09.246461 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.246441    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"]
Sep 17 23:49:09.269520 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.269491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.269652 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.269536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.269652 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.269600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5wln\" (UniqueName: \"kubernetes.io/projected/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kube-api-access-b5wln\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.269757 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.269657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d5138a5-74e9-4a7f-a186-5ba41f48d760-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.269757 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.269691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d5138a5-74e9-4a7f-a186-5ba41f48d760-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.370015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.369995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5wln\" (UniqueName: \"kubernetes.io/projected/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kube-api-access-b5wln\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.370121 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.370022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d5138a5-74e9-4a7f-a186-5ba41f48d760-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.370121 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.370043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d5138a5-74e9-4a7f-a186-5ba41f48d760-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.370234 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:49:09.370176    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 17 23:49:09.370234 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:49:09.370232    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d5138a5-74e9-4a7f-a186-5ba41f48d760-proxy-tls podName:6d5138a5-74e9-4a7f-a186-5ba41f48d760 nodeName:}" failed. No retries permitted until 2026-09-17 23:49:09.870212807 +0000 UTC m=+2282.914588945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6d5138a5-74e9-4a7f-a186-5ba41f48d760-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" (UID: "6d5138a5-74e9-4a7f-a186-5ba41f48d760") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 17 23:49:09.370345 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.370249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.370345 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.370283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.370691 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.370671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.370810 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.370796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d5138a5-74e9-4a7f-a186-5ba41f48d760-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.370866 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.370852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.381056 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.381040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5wln\" (UniqueName: \"kubernetes.io/projected/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kube-api-access-b5wln\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.874302 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.874258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d5138a5-74e9-4a7f-a186-5ba41f48d760-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:09.876989 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:09.876967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d5138a5-74e9-4a7f-a186-5ba41f48d760-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:10.146125 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:10.146037    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:10.416578 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:10.416509    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"]
Sep 17 23:49:10.419565 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:49:10.419537    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5138a5_74e9_4a7f_a186_5ba41f48d760.slice/crio-692864c60413fc972597fd92e32e18a41e60dc35d0ed447b34cde8efacb9489f WatchSource:0}: Error finding container 692864c60413fc972597fd92e32e18a41e60dc35d0ed447b34cde8efacb9489f: Status 404 returned error can't find the container with id 692864c60413fc972597fd92e32e18a41e60dc35d0ed447b34cde8efacb9489f
Sep 17 23:49:11.207799 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:11.207764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" event={"ID":"6d5138a5-74e9-4a7f-a186-5ba41f48d760","Type":"ContainerStarted","Data":"800c0908baaf81eee82c195e1144c6be8630ff3bdf43928c8e85c8da9c037ffa"}
Sep 17 23:49:11.207973 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:11.207819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" event={"ID":"6d5138a5-74e9-4a7f-a186-5ba41f48d760","Type":"ContainerStarted","Data":"692864c60413fc972597fd92e32e18a41e60dc35d0ed447b34cde8efacb9489f"}
Sep 17 23:49:15.222231 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:15.222197    2580 generic.go:356] "Generic (PLEG): container finished" podID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerID="800c0908baaf81eee82c195e1144c6be8630ff3bdf43928c8e85c8da9c037ffa" exitCode=0
Sep 17 23:49:15.222643 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:15.222230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" event={"ID":"6d5138a5-74e9-4a7f-a186-5ba41f48d760","Type":"ContainerDied","Data":"800c0908baaf81eee82c195e1144c6be8630ff3bdf43928c8e85c8da9c037ffa"}
Sep 17 23:49:15.223513 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:15.223498    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:49:22.256520 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:22.256485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" event={"ID":"6d5138a5-74e9-4a7f-a186-5ba41f48d760","Type":"ContainerStarted","Data":"bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70"}
Sep 17 23:49:22.256935 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:22.256547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" event={"ID":"6d5138a5-74e9-4a7f-a186-5ba41f48d760","Type":"ContainerStarted","Data":"2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7"}
Sep 17 23:49:22.256935 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:22.256828    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:22.257020 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:22.256959    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:22.258112 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:22.258086    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 23:49:22.275974 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:22.275914    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podStartSLOduration=3.198768988 podStartE2EDuration="13.275894542s" totalImagesPullingTime="6.114229829s" totalInitContainerRuntime="3.962895725s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:49:22.274579648 +0000 UTC m=+2295.318955818" watchObservedRunningTime="2026-09-17 23:49:22.275894542 +0000 UTC m=+2295.320270701"
Sep 17 23:49:23.259594 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:23.259556    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 23:49:28.264423 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:28.264393    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:49:28.264948 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:28.264918    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 23:49:38.265442 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:38.265401    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 23:49:48.265016 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:48.264979    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 23:49:58.265584 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:49:58.265534    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 23:50:08.265604 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:50:08.265521    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 23:50:18.265769 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:50:18.265723    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 23:50:28.265264 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:50:28.265222    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 23:50:31.496983 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:50:31.496946    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 23:50:41.496913 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:50:41.496866    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 23:50:51.500171 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:50:51.500134    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:51:01.073246 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:01.073210    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"]
Sep 17 23:51:01.073845 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:01.073486    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" containerID="cri-o://2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7" gracePeriod=30
Sep 17 23:51:01.073845 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:01.073589    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kube-rbac-proxy" containerID="cri-o://bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70" gracePeriod=30
Sep 17 23:51:01.497161 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:01.497070    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 23:51:01.586210 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:01.586179    2580 generic.go:356] "Generic (PLEG): container finished" podID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerID="bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70" exitCode=2
Sep 17 23:51:01.586356 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:01.586226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" event={"ID":"6d5138a5-74e9-4a7f-a186-5ba41f48d760","Type":"ContainerDied","Data":"bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70"}
Sep 17 23:51:03.259832 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:03.259788    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 17 23:51:07.313786 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.313763    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:51:07.430083 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.430054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kserve-provision-location\") pod \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") "
Sep 17 23:51:07.430232 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.430100    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d5138a5-74e9-4a7f-a186-5ba41f48d760-cabundle-cert\") pod \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") "
Sep 17 23:51:07.430232 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.430135    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5wln\" (UniqueName: \"kubernetes.io/projected/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kube-api-access-b5wln\") pod \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") "
Sep 17 23:51:07.430232 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.430168    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d5138a5-74e9-4a7f-a186-5ba41f48d760-proxy-tls\") pod \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") "
Sep 17 23:51:07.430232 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.430186    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kube-rbac-proxy-sar-config\") pod \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\" (UID: \"6d5138a5-74e9-4a7f-a186-5ba41f48d760\") "
Sep 17 23:51:07.430460 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.430411    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d5138a5-74e9-4a7f-a186-5ba41f48d760" (UID: "6d5138a5-74e9-4a7f-a186-5ba41f48d760"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:51:07.430521 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.430486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5138a5-74e9-4a7f-a186-5ba41f48d760-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d5138a5-74e9-4a7f-a186-5ba41f48d760" (UID: "6d5138a5-74e9-4a7f-a186-5ba41f48d760"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:51:07.430658 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.430635    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d5138a5-74e9-4a7f-a186-5ba41f48d760" (UID: "6d5138a5-74e9-4a7f-a186-5ba41f48d760"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:51:07.432348 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.432325    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d5138a5-74e9-4a7f-a186-5ba41f48d760-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d5138a5-74e9-4a7f-a186-5ba41f48d760" (UID: "6d5138a5-74e9-4a7f-a186-5ba41f48d760"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:51:07.432420 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.432367    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kube-api-access-b5wln" (OuterVolumeSpecName: "kube-api-access-b5wln") pod "6d5138a5-74e9-4a7f-a186-5ba41f48d760" (UID: "6d5138a5-74e9-4a7f-a186-5ba41f48d760"). InnerVolumeSpecName "kube-api-access-b5wln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:51:07.530720 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.530689    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5wln\" (UniqueName: \"kubernetes.io/projected/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kube-api-access-b5wln\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:51:07.530720 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.530715    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d5138a5-74e9-4a7f-a186-5ba41f48d760-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:51:07.530919 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.530729    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:51:07.530919 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.530742    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d5138a5-74e9-4a7f-a186-5ba41f48d760-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:51:07.530919 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.530750    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d5138a5-74e9-4a7f-a186-5ba41f48d760-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:51:07.608397 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.608367    2580 generic.go:356] "Generic (PLEG): container finished" podID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerID="2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7" exitCode=0
Sep 17 23:51:07.608565 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.608413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" event={"ID":"6d5138a5-74e9-4a7f-a186-5ba41f48d760","Type":"ContainerDied","Data":"2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7"}
Sep 17 23:51:07.608565 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.608435    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"
Sep 17 23:51:07.608565 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.608458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9" event={"ID":"6d5138a5-74e9-4a7f-a186-5ba41f48d760","Type":"ContainerDied","Data":"692864c60413fc972597fd92e32e18a41e60dc35d0ed447b34cde8efacb9489f"}
Sep 17 23:51:07.608565 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.608484    2580 scope.go:117] "RemoveContainer" containerID="bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70"
Sep 17 23:51:07.616098 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.616078    2580 scope.go:117] "RemoveContainer" containerID="2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7"
Sep 17 23:51:07.622943 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.622924    2580 scope.go:117] "RemoveContainer" containerID="800c0908baaf81eee82c195e1144c6be8630ff3bdf43928c8e85c8da9c037ffa"
Sep 17 23:51:07.629717 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.628265    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"]
Sep 17 23:51:07.631757 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.631740    2580 scope.go:117] "RemoveContainer" containerID="bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70"
Sep 17 23:51:07.632324 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:51:07.632205    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70\": container with ID starting with bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70 not found: ID does not exist" containerID="bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70"
Sep 17 23:51:07.632324 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.632256    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70"} err="failed to get container status \"bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70\": rpc error: code = NotFound desc = could not find container \"bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70\": container with ID starting with bf1fadb6bd806120ea1f21f8bdecca8180c6f116a4ceca6e63e7f35b80f70b70 not found: ID does not exist"
Sep 17 23:51:07.632324 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.632280    2580 scope.go:117] "RemoveContainer" containerID="2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7"
Sep 17 23:51:07.632631 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:51:07.632591    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7\": container with ID starting with 2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7 not found: ID does not exist" containerID="2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7"
Sep 17 23:51:07.632699 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.632640    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7"} err="failed to get container status \"2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7\": rpc error: code = NotFound desc = could not find container \"2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7\": container with ID starting with 2e989ea686b003de1621456425853b3213f7c4d85e7d3c45d1f7e750a651b4e7 not found: ID does not exist"
Sep 17 23:51:07.632699 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.632659    2580 scope.go:117] "RemoveContainer" containerID="800c0908baaf81eee82c195e1144c6be8630ff3bdf43928c8e85c8da9c037ffa"
Sep 17 23:51:07.632932 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:51:07.632916    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800c0908baaf81eee82c195e1144c6be8630ff3bdf43928c8e85c8da9c037ffa\": container with ID starting with 800c0908baaf81eee82c195e1144c6be8630ff3bdf43928c8e85c8da9c037ffa not found: ID does not exist" containerID="800c0908baaf81eee82c195e1144c6be8630ff3bdf43928c8e85c8da9c037ffa"
Sep 17 23:51:07.632987 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.632937    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800c0908baaf81eee82c195e1144c6be8630ff3bdf43928c8e85c8da9c037ffa"} err="failed to get container status \"800c0908baaf81eee82c195e1144c6be8630ff3bdf43928c8e85c8da9c037ffa\": rpc error: code = NotFound desc = could not find container \"800c0908baaf81eee82c195e1144c6be8630ff3bdf43928c8e85c8da9c037ffa\": container with ID starting with 800c0908baaf81eee82c195e1144c6be8630ff3bdf43928c8e85c8da9c037ffa not found: ID does not exist"
Sep 17 23:51:07.633345 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:07.633328    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ng9z9"]
Sep 17 23:51:09.500713 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:51:09.500683    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" path="/var/lib/kubelet/pods/6d5138a5-74e9-4a7f-a186-5ba41f48d760/volumes"
Sep 17 23:52:01.542841 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.542809    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"]
Sep 17 23:52:01.543197 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.543039    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container"
Sep 17 23:52:01.543197 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.543049    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container"
Sep 17 23:52:01.543197 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.543064    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kube-rbac-proxy"
Sep 17 23:52:01.543197 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.543072    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kube-rbac-proxy"
Sep 17 23:52:01.543197 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.543090    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="storage-initializer"
Sep 17 23:52:01.543197 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.543098    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="storage-initializer"
Sep 17 23:52:01.543197 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.543138    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kube-rbac-proxy"
Sep 17 23:52:01.543197 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.543145    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d5138a5-74e9-4a7f-a186-5ba41f48d760" containerName="kserve-container"
Sep 17 23:52:01.545966 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.545951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.548523 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.548504    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 17 23:52:01.548653 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.548607    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:52:01.548723 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.548665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:52:01.548723 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.548674    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 23:52:01.548723 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.548675    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:52:01.548723 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.548717    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:52:01.549286 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.549270    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:52:01.555531 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.555514    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"]
Sep 17 23:52:01.707708 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.707678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a943788b-3d5b-4432-acce-1d7d3c37b36b-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.707708 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.707712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a943788b-3d5b-4432-acce-1d7d3c37b36b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.707913 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.707736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a943788b-3d5b-4432-acce-1d7d3c37b36b-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.707913 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.707760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8jht\" (UniqueName: \"kubernetes.io/projected/a943788b-3d5b-4432-acce-1d7d3c37b36b-kube-api-access-w8jht\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.707913 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.707833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a943788b-3d5b-4432-acce-1d7d3c37b36b-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.808244 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.808167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a943788b-3d5b-4432-acce-1d7d3c37b36b-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.808244 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.808201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a943788b-3d5b-4432-acce-1d7d3c37b36b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.808244 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.808222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a943788b-3d5b-4432-acce-1d7d3c37b36b-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.808479 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.808347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jht\" (UniqueName: \"kubernetes.io/projected/a943788b-3d5b-4432-acce-1d7d3c37b36b-kube-api-access-w8jht\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.808479 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.808421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a943788b-3d5b-4432-acce-1d7d3c37b36b-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.808599 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.808581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a943788b-3d5b-4432-acce-1d7d3c37b36b-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.809009 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.808983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a943788b-3d5b-4432-acce-1d7d3c37b36b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.809292 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.808983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a943788b-3d5b-4432-acce-1d7d3c37b36b-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.810876 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.810858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a943788b-3d5b-4432-acce-1d7d3c37b36b-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.816847 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.816823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jht\" (UniqueName: \"kubernetes.io/projected/a943788b-3d5b-4432-acce-1d7d3c37b36b-kube-api-access-w8jht\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:01.855963 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:01.855931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:02.126455 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:02.126349    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"]
Sep 17 23:52:02.129181 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:52:02.129151    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda943788b_3d5b_4432_acce_1d7d3c37b36b.slice/crio-b021d960a9197679ddcb69d50032f2bcbaab0d5c06d86ee35893d8ba45a17777 WatchSource:0}: Error finding container b021d960a9197679ddcb69d50032f2bcbaab0d5c06d86ee35893d8ba45a17777: Status 404 returned error can't find the container with id b021d960a9197679ddcb69d50032f2bcbaab0d5c06d86ee35893d8ba45a17777
Sep 17 23:52:02.796027 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:02.795992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" event={"ID":"a943788b-3d5b-4432-acce-1d7d3c37b36b","Type":"ContainerStarted","Data":"d2ca67c7b79f08cb5f4d3e535ab04cb423fbe9a5d1b5408e56a2f2ba5fb5bf35"}
Sep 17 23:52:02.796399 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:02.796049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" event={"ID":"a943788b-3d5b-4432-acce-1d7d3c37b36b","Type":"ContainerStarted","Data":"b021d960a9197679ddcb69d50032f2bcbaab0d5c06d86ee35893d8ba45a17777"}
Sep 17 23:52:06.811965 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:06.811926    2580 generic.go:356] "Generic (PLEG): container finished" podID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerID="d2ca67c7b79f08cb5f4d3e535ab04cb423fbe9a5d1b5408e56a2f2ba5fb5bf35" exitCode=0
Sep 17 23:52:06.812390 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:06.812014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" event={"ID":"a943788b-3d5b-4432-acce-1d7d3c37b36b","Type":"ContainerDied","Data":"d2ca67c7b79f08cb5f4d3e535ab04cb423fbe9a5d1b5408e56a2f2ba5fb5bf35"}
Sep 17 23:52:07.818593 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:07.818555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" event={"ID":"a943788b-3d5b-4432-acce-1d7d3c37b36b","Type":"ContainerStarted","Data":"39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd"}
Sep 17 23:52:07.819082 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:07.818632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" event={"ID":"a943788b-3d5b-4432-acce-1d7d3c37b36b","Type":"ContainerStarted","Data":"39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1"}
Sep 17 23:52:07.819082 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:07.818822    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:07.837376 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:07.837324    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" podStartSLOduration=2.9791394479999997 podStartE2EDuration="6.837310895s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.858171447s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:52:07.83638841 +0000 UTC m=+2460.880764579" watchObservedRunningTime="2026-09-17 23:52:07.837310895 +0000 UTC m=+2460.881687053"
Sep 17 23:52:08.822446 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:08.822421    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:08.823970 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:08.823935    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 23:52:09.826202 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:09.826165    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 23:52:14.830995 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:14.830968    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:14.831670 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:14.831636    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 23:52:22.279546 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.279513    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"]
Sep 17 23:52:22.282780 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.282759    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.285114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.285095    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 23:52:22.285389 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.285373    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 17 23:52:22.295968 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.295939    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"]
Sep 17 23:52:22.354548 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.354519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15648021-99eb-41a4-b007-b475377253e2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.354772 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.354559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15648021-99eb-41a4-b007-b475377253e2-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.354772 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.354577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ln4h\" (UniqueName: \"kubernetes.io/projected/15648021-99eb-41a4-b007-b475377253e2-kube-api-access-4ln4h\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.354772 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.354600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15648021-99eb-41a4-b007-b475377253e2-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.354772 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.354694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15648021-99eb-41a4-b007-b475377253e2-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.455380 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.455347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15648021-99eb-41a4-b007-b475377253e2-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.455380 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.455386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15648021-99eb-41a4-b007-b475377253e2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.455657 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.455412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15648021-99eb-41a4-b007-b475377253e2-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.455657 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.455439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ln4h\" (UniqueName: \"kubernetes.io/projected/15648021-99eb-41a4-b007-b475377253e2-kube-api-access-4ln4h\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.455657 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.455472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15648021-99eb-41a4-b007-b475377253e2-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.455870 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.455851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15648021-99eb-41a4-b007-b475377253e2-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.456220 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.456194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15648021-99eb-41a4-b007-b475377253e2-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.456323 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.456194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15648021-99eb-41a4-b007-b475377253e2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.458035 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.458012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15648021-99eb-41a4-b007-b475377253e2-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.465954 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.465936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ln4h\" (UniqueName: \"kubernetes.io/projected/15648021-99eb-41a4-b007-b475377253e2-kube-api-access-4ln4h\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w9xrn\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.594543 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.594503    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:22.866146 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:22.866073    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"]
Sep 17 23:52:22.870285 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:52:22.870257    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15648021_99eb_41a4_b007_b475377253e2.slice/crio-da336359c1be9b45585284bbdaf31c4d7440327b288dc17bfb5a61ed2f7a63cf WatchSource:0}: Error finding container da336359c1be9b45585284bbdaf31c4d7440327b288dc17bfb5a61ed2f7a63cf: Status 404 returned error can't find the container with id da336359c1be9b45585284bbdaf31c4d7440327b288dc17bfb5a61ed2f7a63cf
Sep 17 23:52:23.878555 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:23.878523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" event={"ID":"15648021-99eb-41a4-b007-b475377253e2","Type":"ContainerStarted","Data":"8b199e2cb836138f38f7aeec02fb0d862bbb30067de58fca295d69cbec12d024"}
Sep 17 23:52:23.878942 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:23.878574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" event={"ID":"15648021-99eb-41a4-b007-b475377253e2","Type":"ContainerStarted","Data":"da336359c1be9b45585284bbdaf31c4d7440327b288dc17bfb5a61ed2f7a63cf"}
Sep 17 23:52:24.832351 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:24.832328    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:27.894870 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:27.894840    2580 generic.go:356] "Generic (PLEG): container finished" podID="15648021-99eb-41a4-b007-b475377253e2" containerID="8b199e2cb836138f38f7aeec02fb0d862bbb30067de58fca295d69cbec12d024" exitCode=0
Sep 17 23:52:27.895212 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:27.894915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" event={"ID":"15648021-99eb-41a4-b007-b475377253e2","Type":"ContainerDied","Data":"8b199e2cb836138f38f7aeec02fb0d862bbb30067de58fca295d69cbec12d024"}
Sep 17 23:52:28.900178 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:28.900146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" event={"ID":"15648021-99eb-41a4-b007-b475377253e2","Type":"ContainerStarted","Data":"17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf"}
Sep 17 23:52:28.900570 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:28.900205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" event={"ID":"15648021-99eb-41a4-b007-b475377253e2","Type":"ContainerStarted","Data":"a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0"}
Sep 17 23:52:28.900570 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:28.900475    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:28.900703 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:28.900648    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:28.901854 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:28.901827    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 23:52:28.917601 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:28.917559    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podStartSLOduration=2.904808562 podStartE2EDuration="6.917544558s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.012735996s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:52:28.917086947 +0000 UTC m=+2481.961463118" watchObservedRunningTime="2026-09-17 23:52:28.917544558 +0000 UTC m=+2481.961920718"
Sep 17 23:52:29.903801 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:29.903763    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 23:52:31.648779 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:31.648750    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"]
Sep 17 23:52:31.649167 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:31.648998    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="kserve-container" containerID="cri-o://39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1" gracePeriod=30
Sep 17 23:52:31.649167 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:31.649089    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="kube-rbac-proxy" containerID="cri-o://39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd" gracePeriod=30
Sep 17 23:52:31.918003 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:31.917912    2580 generic.go:356] "Generic (PLEG): container finished" podID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerID="39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd" exitCode=2
Sep 17 23:52:31.918144 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:31.917993    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" event={"ID":"a943788b-3d5b-4432-acce-1d7d3c37b36b","Type":"ContainerDied","Data":"39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd"}
Sep 17 23:52:32.382301 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.382280    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:32.431808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.431778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a943788b-3d5b-4432-acce-1d7d3c37b36b-cabundle-cert\") pod \"a943788b-3d5b-4432-acce-1d7d3c37b36b\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") "
Sep 17 23:52:32.431808 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.431809    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a943788b-3d5b-4432-acce-1d7d3c37b36b-kserve-provision-location\") pod \"a943788b-3d5b-4432-acce-1d7d3c37b36b\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") "
Sep 17 23:52:32.432011 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.431838    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a943788b-3d5b-4432-acce-1d7d3c37b36b-kube-rbac-proxy-sar-config\") pod \"a943788b-3d5b-4432-acce-1d7d3c37b36b\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") "
Sep 17 23:52:32.432011 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.431857    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8jht\" (UniqueName: \"kubernetes.io/projected/a943788b-3d5b-4432-acce-1d7d3c37b36b-kube-api-access-w8jht\") pod \"a943788b-3d5b-4432-acce-1d7d3c37b36b\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") "
Sep 17 23:52:32.432011 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.431937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a943788b-3d5b-4432-acce-1d7d3c37b36b-proxy-tls\") pod \"a943788b-3d5b-4432-acce-1d7d3c37b36b\" (UID: \"a943788b-3d5b-4432-acce-1d7d3c37b36b\") "
Sep 17 23:52:32.432173 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.432150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a943788b-3d5b-4432-acce-1d7d3c37b36b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a943788b-3d5b-4432-acce-1d7d3c37b36b" (UID: "a943788b-3d5b-4432-acce-1d7d3c37b36b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:52:32.432173 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.432163    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a943788b-3d5b-4432-acce-1d7d3c37b36b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a943788b-3d5b-4432-acce-1d7d3c37b36b" (UID: "a943788b-3d5b-4432-acce-1d7d3c37b36b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:52:32.432253 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.432165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a943788b-3d5b-4432-acce-1d7d3c37b36b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a943788b-3d5b-4432-acce-1d7d3c37b36b" (UID: "a943788b-3d5b-4432-acce-1d7d3c37b36b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:52:32.433992 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.433971    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a943788b-3d5b-4432-acce-1d7d3c37b36b-kube-api-access-w8jht" (OuterVolumeSpecName: "kube-api-access-w8jht") pod "a943788b-3d5b-4432-acce-1d7d3c37b36b" (UID: "a943788b-3d5b-4432-acce-1d7d3c37b36b"). InnerVolumeSpecName "kube-api-access-w8jht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:52:32.434077 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.434056    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a943788b-3d5b-4432-acce-1d7d3c37b36b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a943788b-3d5b-4432-acce-1d7d3c37b36b" (UID: "a943788b-3d5b-4432-acce-1d7d3c37b36b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:52:32.532789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.532754    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a943788b-3d5b-4432-acce-1d7d3c37b36b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:52:32.532789 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.532788    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8jht\" (UniqueName: \"kubernetes.io/projected/a943788b-3d5b-4432-acce-1d7d3c37b36b-kube-api-access-w8jht\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:52:32.532982 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.532802    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a943788b-3d5b-4432-acce-1d7d3c37b36b-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:52:32.532982 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.532815    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a943788b-3d5b-4432-acce-1d7d3c37b36b-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:52:32.532982 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.532826    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a943788b-3d5b-4432-acce-1d7d3c37b36b-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:52:32.923372 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.923274    2580 generic.go:356] "Generic (PLEG): container finished" podID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerID="39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1" exitCode=0
Sep 17 23:52:32.923372 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.923328    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"
Sep 17 23:52:32.923372 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.923347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" event={"ID":"a943788b-3d5b-4432-acce-1d7d3c37b36b","Type":"ContainerDied","Data":"39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1"}
Sep 17 23:52:32.923878 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.923405    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq" event={"ID":"a943788b-3d5b-4432-acce-1d7d3c37b36b","Type":"ContainerDied","Data":"b021d960a9197679ddcb69d50032f2bcbaab0d5c06d86ee35893d8ba45a17777"}
Sep 17 23:52:32.923878 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.923433    2580 scope.go:117] "RemoveContainer" containerID="39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd"
Sep 17 23:52:32.931733 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.931711    2580 scope.go:117] "RemoveContainer" containerID="39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1"
Sep 17 23:52:32.938674 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.938654    2580 scope.go:117] "RemoveContainer" containerID="d2ca67c7b79f08cb5f4d3e535ab04cb423fbe9a5d1b5408e56a2f2ba5fb5bf35"
Sep 17 23:52:32.945932 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.945916    2580 scope.go:117] "RemoveContainer" containerID="39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd"
Sep 17 23:52:32.945986 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.945949    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"]
Sep 17 23:52:32.946173 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:52:32.946151    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd\": container with ID starting with 39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd not found: ID does not exist" containerID="39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd"
Sep 17 23:52:32.946215 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.946182    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd"} err="failed to get container status \"39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd\": rpc error: code = NotFound desc = could not find container \"39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd\": container with ID starting with 39062e9dcd633d095a506c0ae713e8873e7f5d6ef8408b569ddfff3c7e370cfd not found: ID does not exist"
Sep 17 23:52:32.946215 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.946201    2580 scope.go:117] "RemoveContainer" containerID="39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1"
Sep 17 23:52:32.946388 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:52:32.946373    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1\": container with ID starting with 39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1 not found: ID does not exist" containerID="39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1"
Sep 17 23:52:32.946424 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.946394    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1"} err="failed to get container status \"39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1\": rpc error: code = NotFound desc = could not find container \"39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1\": container with ID starting with 39626aa222888e05bbf8ced7488014b980d94e362088eb835ec0533ed7726bb1 not found: ID does not exist"
Sep 17 23:52:32.946424 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.946407    2580 scope.go:117] "RemoveContainer" containerID="d2ca67c7b79f08cb5f4d3e535ab04cb423fbe9a5d1b5408e56a2f2ba5fb5bf35"
Sep 17 23:52:32.946606 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:52:32.946592    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ca67c7b79f08cb5f4d3e535ab04cb423fbe9a5d1b5408e56a2f2ba5fb5bf35\": container with ID starting with d2ca67c7b79f08cb5f4d3e535ab04cb423fbe9a5d1b5408e56a2f2ba5fb5bf35 not found: ID does not exist" containerID="d2ca67c7b79f08cb5f4d3e535ab04cb423fbe9a5d1b5408e56a2f2ba5fb5bf35"
Sep 17 23:52:32.946683 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.946632    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ca67c7b79f08cb5f4d3e535ab04cb423fbe9a5d1b5408e56a2f2ba5fb5bf35"} err="failed to get container status \"d2ca67c7b79f08cb5f4d3e535ab04cb423fbe9a5d1b5408e56a2f2ba5fb5bf35\": rpc error: code = NotFound desc = could not find container \"d2ca67c7b79f08cb5f4d3e535ab04cb423fbe9a5d1b5408e56a2f2ba5fb5bf35\": container with ID starting with d2ca67c7b79f08cb5f4d3e535ab04cb423fbe9a5d1b5408e56a2f2ba5fb5bf35 not found: ID does not exist"
Sep 17 23:52:32.949356 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:32.949338    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-t6fjq"]
Sep 17 23:52:33.500278 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:33.500246    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" path="/var/lib/kubelet/pods/a943788b-3d5b-4432-acce-1d7d3c37b36b/volumes"
Sep 17 23:52:34.908130 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:34.908106    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:52:34.908654 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:34.908628    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 23:52:44.908944 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:44.908905    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 23:52:54.909484 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:52:54.909440    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 23:53:04.908864 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:04.908826    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 23:53:14.909042 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:14.909000    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 23:53:24.908602 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:24.908564    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 23:53:32.116608 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.116571    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"]
Sep 17 23:53:32.116992 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.116870    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="kube-rbac-proxy"
Sep 17 23:53:32.116992 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.116883    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="kube-rbac-proxy"
Sep 17 23:53:32.116992 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.116892    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="storage-initializer"
Sep 17 23:53:32.116992 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.116898    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="storage-initializer"
Sep 17 23:53:32.116992 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.116907    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="kserve-container"
Sep 17 23:53:32.116992 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.116913    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="kserve-container"
Sep 17 23:53:32.116992 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.116952    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="kserve-container"
Sep 17 23:53:32.116992 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.116960    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a943788b-3d5b-4432-acce-1d7d3c37b36b" containerName="kube-rbac-proxy"
Sep 17 23:53:32.120051 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.120036    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.122137 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.122113    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 17 23:53:32.122277 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.122187    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 17 23:53:32.131012 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.130991    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"]
Sep 17 23:53:32.262678 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.262633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40cd72b-a633-4836-b616-aa0a15598e44-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.262678 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.262683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40cd72b-a633-4836-b616-aa0a15598e44-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.262902 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.262700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40cd72b-a633-4836-b616-aa0a15598e44-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.262902 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.262765    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8mg8\" (UniqueName: \"kubernetes.io/projected/c40cd72b-a633-4836-b616-aa0a15598e44-kube-api-access-n8mg8\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.262902 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.262817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40cd72b-a633-4836-b616-aa0a15598e44-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.363896 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.363859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40cd72b-a633-4836-b616-aa0a15598e44-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.363896 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.363896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40cd72b-a633-4836-b616-aa0a15598e44-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.364112 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.363938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8mg8\" (UniqueName: \"kubernetes.io/projected/c40cd72b-a633-4836-b616-aa0a15598e44-kube-api-access-n8mg8\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.364153 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.364114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40cd72b-a633-4836-b616-aa0a15598e44-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.364217 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.364198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40cd72b-a633-4836-b616-aa0a15598e44-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.364471 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.364444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40cd72b-a633-4836-b616-aa0a15598e44-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.364723 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.364704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40cd72b-a633-4836-b616-aa0a15598e44-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.364796 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.364782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40cd72b-a633-4836-b616-aa0a15598e44-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.366754 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.366697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40cd72b-a633-4836-b616-aa0a15598e44-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.372725 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.372706    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8mg8\" (UniqueName: \"kubernetes.io/projected/c40cd72b-a633-4836-b616-aa0a15598e44-kube-api-access-n8mg8\") pod \"isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.430898 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.430862    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:32.705357 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:32.705282    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"]
Sep 17 23:53:32.708171 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:53:32.708144    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40cd72b_a633_4836_b616_aa0a15598e44.slice/crio-ccf63d0e23d96a46ffad2c00958a24395eb692ad9e44b4fbc439b3f0bb1ee38b WatchSource:0}: Error finding container ccf63d0e23d96a46ffad2c00958a24395eb692ad9e44b4fbc439b3f0bb1ee38b: Status 404 returned error can't find the container with id ccf63d0e23d96a46ffad2c00958a24395eb692ad9e44b4fbc439b3f0bb1ee38b
Sep 17 23:53:33.137787 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:33.137743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" event={"ID":"c40cd72b-a633-4836-b616-aa0a15598e44","Type":"ContainerStarted","Data":"1b65799cb29aebebdccb7d1b6fee0de1942ccc192a81ec4baafdbc4c7fed5186"}
Sep 17 23:53:33.138256 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:33.137839    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" event={"ID":"c40cd72b-a633-4836-b616-aa0a15598e44","Type":"ContainerStarted","Data":"ccf63d0e23d96a46ffad2c00958a24395eb692ad9e44b4fbc439b3f0bb1ee38b"}
Sep 17 23:53:34.909772 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:34.909739    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:53:37.152985 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:37.152960    2580 generic.go:356] "Generic (PLEG): container finished" podID="c40cd72b-a633-4836-b616-aa0a15598e44" containerID="1b65799cb29aebebdccb7d1b6fee0de1942ccc192a81ec4baafdbc4c7fed5186" exitCode=0
Sep 17 23:53:37.153371 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:37.153002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" event={"ID":"c40cd72b-a633-4836-b616-aa0a15598e44","Type":"ContainerDied","Data":"1b65799cb29aebebdccb7d1b6fee0de1942ccc192a81ec4baafdbc4c7fed5186"}
Sep 17 23:53:38.160131 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:38.160037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" event={"ID":"c40cd72b-a633-4836-b616-aa0a15598e44","Type":"ContainerStarted","Data":"a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b"}
Sep 17 23:53:40.170982 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:40.170949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" event={"ID":"c40cd72b-a633-4836-b616-aa0a15598e44","Type":"ContainerStarted","Data":"da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1"}
Sep 17 23:53:41.177254 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:41.177217    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" event={"ID":"c40cd72b-a633-4836-b616-aa0a15598e44","Type":"ContainerStarted","Data":"7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e"}
Sep 17 23:53:41.177656 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:41.177469    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:41.177656 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:41.177527    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:41.199669 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:41.199603    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podStartSLOduration=2.439529133 podStartE2EDuration="9.199572242s" totalImagesPullingTime="2.878385706s" totalInitContainerRuntime="3.881657403s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:53:41.197188357 +0000 UTC m=+2554.241564529" watchObservedRunningTime="2026-09-17 23:53:41.199572242 +0000 UTC m=+2554.243948400"
Sep 17 23:53:42.180918 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:42.180894    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:53:42.878602 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:42.878570    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"]
Sep 17 23:53:42.878841 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:42.878818    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container" containerID="cri-o://a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0" gracePeriod=30
Sep 17 23:53:42.878924 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:42.878849    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kube-rbac-proxy" containerID="cri-o://17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf" gracePeriod=30
Sep 17 23:53:43.187678 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:43.187588    2580 generic.go:356] "Generic (PLEG): container finished" podID="15648021-99eb-41a4-b007-b475377253e2" containerID="17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf" exitCode=2
Sep 17 23:53:43.187678 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:43.187657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" event={"ID":"15648021-99eb-41a4-b007-b475377253e2","Type":"ContainerDied","Data":"17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf"}
Sep 17 23:53:44.904133 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:44.904097    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 23:53:44.909016 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:44.908992    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 23:53:47.021357 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.021332    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:53:47.079892 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.079867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15648021-99eb-41a4-b007-b475377253e2-kube-rbac-proxy-sar-config\") pod \"15648021-99eb-41a4-b007-b475377253e2\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") "
Sep 17 23:53:47.080065 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.079908    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ln4h\" (UniqueName: \"kubernetes.io/projected/15648021-99eb-41a4-b007-b475377253e2-kube-api-access-4ln4h\") pod \"15648021-99eb-41a4-b007-b475377253e2\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") "
Sep 17 23:53:47.080065 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.079952    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15648021-99eb-41a4-b007-b475377253e2-kserve-provision-location\") pod \"15648021-99eb-41a4-b007-b475377253e2\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") "
Sep 17 23:53:47.080065 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.079979    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15648021-99eb-41a4-b007-b475377253e2-proxy-tls\") pod \"15648021-99eb-41a4-b007-b475377253e2\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") "
Sep 17 23:53:47.080065 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.080011    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15648021-99eb-41a4-b007-b475377253e2-cabundle-cert\") pod \"15648021-99eb-41a4-b007-b475377253e2\" (UID: \"15648021-99eb-41a4-b007-b475377253e2\") "
Sep 17 23:53:47.080281 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.080209    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15648021-99eb-41a4-b007-b475377253e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15648021-99eb-41a4-b007-b475377253e2" (UID: "15648021-99eb-41a4-b007-b475377253e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:53:47.080399 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.080370    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15648021-99eb-41a4-b007-b475377253e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15648021-99eb-41a4-b007-b475377253e2" (UID: "15648021-99eb-41a4-b007-b475377253e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:53:47.080399 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.080388    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15648021-99eb-41a4-b007-b475377253e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "15648021-99eb-41a4-b007-b475377253e2" (UID: "15648021-99eb-41a4-b007-b475377253e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:53:47.082193 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.082162    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15648021-99eb-41a4-b007-b475377253e2-kube-api-access-4ln4h" (OuterVolumeSpecName: "kube-api-access-4ln4h") pod "15648021-99eb-41a4-b007-b475377253e2" (UID: "15648021-99eb-41a4-b007-b475377253e2"). InnerVolumeSpecName "kube-api-access-4ln4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:53:47.082193 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.082169    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15648021-99eb-41a4-b007-b475377253e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15648021-99eb-41a4-b007-b475377253e2" (UID: "15648021-99eb-41a4-b007-b475377253e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:53:47.180890 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.180805    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15648021-99eb-41a4-b007-b475377253e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:53:47.180890 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.180834    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ln4h\" (UniqueName: \"kubernetes.io/projected/15648021-99eb-41a4-b007-b475377253e2-kube-api-access-4ln4h\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:53:47.180890 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.180846    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15648021-99eb-41a4-b007-b475377253e2-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:53:47.180890 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.180858    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15648021-99eb-41a4-b007-b475377253e2-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:53:47.180890 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.180870    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15648021-99eb-41a4-b007-b475377253e2-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:53:47.203147 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.203107    2580 generic.go:356] "Generic (PLEG): container finished" podID="15648021-99eb-41a4-b007-b475377253e2" containerID="a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0" exitCode=0
Sep 17 23:53:47.203253 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.203163    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"
Sep 17 23:53:47.203253 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.203171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" event={"ID":"15648021-99eb-41a4-b007-b475377253e2","Type":"ContainerDied","Data":"a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0"}
Sep 17 23:53:47.203345 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.203257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn" event={"ID":"15648021-99eb-41a4-b007-b475377253e2","Type":"ContainerDied","Data":"da336359c1be9b45585284bbdaf31c4d7440327b288dc17bfb5a61ed2f7a63cf"}
Sep 17 23:53:47.203345 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.203284    2580 scope.go:117] "RemoveContainer" containerID="17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf"
Sep 17 23:53:47.211889 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.211870    2580 scope.go:117] "RemoveContainer" containerID="a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0"
Sep 17 23:53:47.218703 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.218687    2580 scope.go:117] "RemoveContainer" containerID="8b199e2cb836138f38f7aeec02fb0d862bbb30067de58fca295d69cbec12d024"
Sep 17 23:53:47.223754 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.223736    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"]
Sep 17 23:53:47.225800 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.225787    2580 scope.go:117] "RemoveContainer" containerID="17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf"
Sep 17 23:53:47.226078 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:53:47.226059    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf\": container with ID starting with 17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf not found: ID does not exist" containerID="17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf"
Sep 17 23:53:47.226166 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.226089    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf"} err="failed to get container status \"17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf\": rpc error: code = NotFound desc = could not find container \"17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf\": container with ID starting with 17abe1837fa5313e223fed672f6aad186cd84f341f25f07e58efc5965b069cdf not found: ID does not exist"
Sep 17 23:53:47.226166 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.226114    2580 scope.go:117] "RemoveContainer" containerID="a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0"
Sep 17 23:53:47.226796 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:53:47.226550    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0\": container with ID starting with a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0 not found: ID does not exist" containerID="a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0"
Sep 17 23:53:47.226796 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.226585    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0"} err="failed to get container status \"a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0\": rpc error: code = NotFound desc = could not find container \"a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0\": container with ID starting with a8b0bf22d52d635dfa0a961a2fc67a22489e36c888bbbb8c1d67cc9fb43507c0 not found: ID does not exist"
Sep 17 23:53:47.226796 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.226607    2580 scope.go:117] "RemoveContainer" containerID="8b199e2cb836138f38f7aeec02fb0d862bbb30067de58fca295d69cbec12d024"
Sep 17 23:53:47.227037 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:53:47.227021    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b199e2cb836138f38f7aeec02fb0d862bbb30067de58fca295d69cbec12d024\": container with ID starting with 8b199e2cb836138f38f7aeec02fb0d862bbb30067de58fca295d69cbec12d024 not found: ID does not exist" containerID="8b199e2cb836138f38f7aeec02fb0d862bbb30067de58fca295d69cbec12d024"
Sep 17 23:53:47.227085 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.227044    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b199e2cb836138f38f7aeec02fb0d862bbb30067de58fca295d69cbec12d024"} err="failed to get container status \"8b199e2cb836138f38f7aeec02fb0d862bbb30067de58fca295d69cbec12d024\": rpc error: code = NotFound desc = could not find container \"8b199e2cb836138f38f7aeec02fb0d862bbb30067de58fca295d69cbec12d024\": container with ID starting with 8b199e2cb836138f38f7aeec02fb0d862bbb30067de58fca295d69cbec12d024 not found: ID does not exist"
Sep 17 23:53:47.228000 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.227985    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w9xrn"]
Sep 17 23:53:47.500584 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:47.500507    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15648021-99eb-41a4-b007-b475377253e2" path="/var/lib/kubelet/pods/15648021-99eb-41a4-b007-b475377253e2/volumes"
Sep 17 23:53:48.192968 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:53:48.192942    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:54:08.193827 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:08.193790    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.41:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 23:54:18.194932 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:18.194901    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:54:48.195797 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:48.195726    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:54:52.176942 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:52.176912    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"]
Sep 17 23:54:52.177367 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:52.177138    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kserve-container" containerID="cri-o://a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b" gracePeriod=30
Sep 17 23:54:52.177367 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:52.177200    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kserve-agent" containerID="cri-o://da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1" gracePeriod=30
Sep 17 23:54:52.177367 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:52.177244    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kube-rbac-proxy" containerID="cri-o://7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e" gracePeriod=30
Sep 17 23:54:52.436744 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:52.436658    2580 generic.go:356] "Generic (PLEG): container finished" podID="c40cd72b-a633-4836-b616-aa0a15598e44" containerID="7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e" exitCode=2
Sep 17 23:54:52.436744 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:52.436696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" event={"ID":"c40cd72b-a633-4836-b616-aa0a15598e44","Type":"ContainerDied","Data":"7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e"}
Sep 17 23:54:53.188258 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:53.188220    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 23:54:54.445539 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:54.445505    2580 generic.go:356] "Generic (PLEG): container finished" podID="c40cd72b-a633-4836-b616-aa0a15598e44" containerID="a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b" exitCode=0
Sep 17 23:54:54.445916 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:54.445554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" event={"ID":"c40cd72b-a633-4836-b616-aa0a15598e44","Type":"ContainerDied","Data":"a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b"}
Sep 17 23:54:58.188383 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:58.188344    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 23:54:58.193800 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:54:58.193765    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.41:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 23:55:03.188868 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:03.188828    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 23:55:03.189248 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:03.189010    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:55:08.188699 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.188653    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 23:55:08.194105 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.194074    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.41:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 23:55:08.453079 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.452998    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"]
Sep 17 23:55:08.453275 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.453263    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="storage-initializer"
Sep 17 23:55:08.453318 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.453276    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="storage-initializer"
Sep 17 23:55:08.453318 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.453292    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container"
Sep 17 23:55:08.453318 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.453297    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container"
Sep 17 23:55:08.453318 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.453306    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kube-rbac-proxy"
Sep 17 23:55:08.453318 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.453313    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kube-rbac-proxy"
Sep 17 23:55:08.453497 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.453353    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kserve-container"
Sep 17 23:55:08.453497 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.453361    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15648021-99eb-41a4-b007-b475377253e2" containerName="kube-rbac-proxy"
Sep 17 23:55:08.456308 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.456291    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.458503 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.458481    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 17 23:55:08.458648 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.458490    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 23:55:08.458804 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.458786    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 17 23:55:08.465325 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.465291    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"]
Sep 17 23:55:08.572058 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.572019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vnmq\" (UniqueName: \"kubernetes.io/projected/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kube-api-access-5vnmq\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.572058 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.572060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.572273 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.572126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e3f7233-4a4d-4274-8bf8-919e9f59608a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.572273 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.572204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e3f7233-4a4d-4274-8bf8-919e9f59608a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.572273 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.572230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.672686 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.672651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e3f7233-4a4d-4274-8bf8-919e9f59608a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.672874 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.672703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e3f7233-4a4d-4274-8bf8-919e9f59608a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.672874 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.672738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.672874 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.672764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vnmq\" (UniqueName: \"kubernetes.io/projected/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kube-api-access-5vnmq\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.673049 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.672883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.673277 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.673253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.673384 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.673300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e3f7233-4a4d-4274-8bf8-919e9f59608a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.673453 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.673402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.675375 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.675355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e3f7233-4a4d-4274-8bf8-919e9f59608a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.681680 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.681657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vnmq\" (UniqueName: \"kubernetes.io/projected/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kube-api-access-5vnmq\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:08.767455 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:08.767362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:09.040311 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:09.040278    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"]
Sep 17 23:55:09.043104 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:55:09.043076    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3f7233_4a4d_4274_8bf8_919e9f59608a.slice/crio-0bc6c115edcb53ce663a0b213f0ddb4bd2c49444d8b817727022563fb0bebc9a WatchSource:0}: Error finding container 0bc6c115edcb53ce663a0b213f0ddb4bd2c49444d8b817727022563fb0bebc9a: Status 404 returned error can't find the container with id 0bc6c115edcb53ce663a0b213f0ddb4bd2c49444d8b817727022563fb0bebc9a
Sep 17 23:55:09.501760 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:09.501675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" event={"ID":"7e3f7233-4a4d-4274-8bf8-919e9f59608a","Type":"ContainerStarted","Data":"ed3750dc49303bfbf4c778ffb90c8bda588da1ca6424c65597b9861edb1ecaaf"}
Sep 17 23:55:09.501760 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:09.501736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" event={"ID":"7e3f7233-4a4d-4274-8bf8-919e9f59608a","Type":"ContainerStarted","Data":"0bc6c115edcb53ce663a0b213f0ddb4bd2c49444d8b817727022563fb0bebc9a"}
Sep 17 23:55:10.504277 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:10.504237    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerID="ed3750dc49303bfbf4c778ffb90c8bda588da1ca6424c65597b9861edb1ecaaf" exitCode=0
Sep 17 23:55:10.504711 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:10.504326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" event={"ID":"7e3f7233-4a4d-4274-8bf8-919e9f59608a","Type":"ContainerDied","Data":"ed3750dc49303bfbf4c778ffb90c8bda588da1ca6424c65597b9861edb1ecaaf"}
Sep 17 23:55:11.510182 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:11.510145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" event={"ID":"7e3f7233-4a4d-4274-8bf8-919e9f59608a","Type":"ContainerStarted","Data":"cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826"}
Sep 17 23:55:11.510654 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:11.510204    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:11.510654 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:11.510224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" event={"ID":"7e3f7233-4a4d-4274-8bf8-919e9f59608a","Type":"ContainerStarted","Data":"0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d"}
Sep 17 23:55:11.510654 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:11.510243    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:11.511853 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:11.511828    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:55:12.514261 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:12.514224    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:55:13.188776 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:13.188732    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 23:55:17.518764 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:17.518738    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:55:17.519221 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:17.519198    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:55:17.538520 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:17.538477    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podStartSLOduration=8.278497463 podStartE2EDuration="9.538464231s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.259966768s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:55:11.527826777 +0000 UTC m=+2644.572202937" watchObservedRunningTime="2026-09-17 23:55:17.538464231 +0000 UTC m=+2650.582840390"
Sep 17 23:55:18.188585 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:18.188548    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 23:55:18.193461 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:18.193427    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.41:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 23:55:18.193567 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:18.193544    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:55:22.365716 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.365687    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:55:22.477853 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.477761    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40cd72b-a633-4836-b616-aa0a15598e44-proxy-tls\") pod \"c40cd72b-a633-4836-b616-aa0a15598e44\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") "
Sep 17 23:55:22.477853 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.477803    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40cd72b-a633-4836-b616-aa0a15598e44-kserve-provision-location\") pod \"c40cd72b-a633-4836-b616-aa0a15598e44\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") "
Sep 17 23:55:22.477853 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.477832    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40cd72b-a633-4836-b616-aa0a15598e44-cabundle-cert\") pod \"c40cd72b-a633-4836-b616-aa0a15598e44\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") "
Sep 17 23:55:22.477853 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.477855    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40cd72b-a633-4836-b616-aa0a15598e44-kube-rbac-proxy-sar-config\") pod \"c40cd72b-a633-4836-b616-aa0a15598e44\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") "
Sep 17 23:55:22.478201 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.477928    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8mg8\" (UniqueName: \"kubernetes.io/projected/c40cd72b-a633-4836-b616-aa0a15598e44-kube-api-access-n8mg8\") pod \"c40cd72b-a633-4836-b616-aa0a15598e44\" (UID: \"c40cd72b-a633-4836-b616-aa0a15598e44\") "
Sep 17 23:55:22.478201 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.478116    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40cd72b-a633-4836-b616-aa0a15598e44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c40cd72b-a633-4836-b616-aa0a15598e44" (UID: "c40cd72b-a633-4836-b616-aa0a15598e44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:55:22.478287 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.478240    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40cd72b-a633-4836-b616-aa0a15598e44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c40cd72b-a633-4836-b616-aa0a15598e44" (UID: "c40cd72b-a633-4836-b616-aa0a15598e44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:55:22.478287 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.478273    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40cd72b-a633-4836-b616-aa0a15598e44-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c40cd72b-a633-4836-b616-aa0a15598e44" (UID: "c40cd72b-a633-4836-b616-aa0a15598e44"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:55:22.480119 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.480096    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40cd72b-a633-4836-b616-aa0a15598e44-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c40cd72b-a633-4836-b616-aa0a15598e44" (UID: "c40cd72b-a633-4836-b616-aa0a15598e44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:55:22.480239 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.480217    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40cd72b-a633-4836-b616-aa0a15598e44-kube-api-access-n8mg8" (OuterVolumeSpecName: "kube-api-access-n8mg8") pod "c40cd72b-a633-4836-b616-aa0a15598e44" (UID: "c40cd72b-a633-4836-b616-aa0a15598e44"). InnerVolumeSpecName "kube-api-access-n8mg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:55:22.551894 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.551862    2580 generic.go:356] "Generic (PLEG): container finished" podID="c40cd72b-a633-4836-b616-aa0a15598e44" containerID="da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1" exitCode=137
Sep 17 23:55:22.552087 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.551912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" event={"ID":"c40cd72b-a633-4836-b616-aa0a15598e44","Type":"ContainerDied","Data":"da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1"}
Sep 17 23:55:22.552087 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.551931    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"
Sep 17 23:55:22.552087 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.551973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn" event={"ID":"c40cd72b-a633-4836-b616-aa0a15598e44","Type":"ContainerDied","Data":"ccf63d0e23d96a46ffad2c00958a24395eb692ad9e44b4fbc439b3f0bb1ee38b"}
Sep 17 23:55:22.552087 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.552004    2580 scope.go:117] "RemoveContainer" containerID="7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e"
Sep 17 23:55:22.559990 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.559965    2580 scope.go:117] "RemoveContainer" containerID="da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1"
Sep 17 23:55:22.567224 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.567198    2580 scope.go:117] "RemoveContainer" containerID="a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b"
Sep 17 23:55:22.574319 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.574297    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"]
Sep 17 23:55:22.574724 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.574710    2580 scope.go:117] "RemoveContainer" containerID="1b65799cb29aebebdccb7d1b6fee0de1942ccc192a81ec4baafdbc4c7fed5186"
Sep 17 23:55:22.578999 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.578973    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8mg8\" (UniqueName: \"kubernetes.io/projected/c40cd72b-a633-4836-b616-aa0a15598e44-kube-api-access-n8mg8\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:55:22.579109 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.579003    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c40cd72b-a633-4836-b616-aa0a15598e44-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:55:22.579109 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.579018    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c40cd72b-a633-4836-b616-aa0a15598e44-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:55:22.579109 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.579032    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c40cd72b-a633-4836-b616-aa0a15598e44-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:55:22.579109 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.579045    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c40cd72b-a633-4836-b616-aa0a15598e44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:55:22.580700 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.580679    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5955bd5d6f-c9bsn"]
Sep 17 23:55:22.582504 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.582474    2580 scope.go:117] "RemoveContainer" containerID="7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e"
Sep 17 23:55:22.582792 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:55:22.582768    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e\": container with ID starting with 7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e not found: ID does not exist" containerID="7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e"
Sep 17 23:55:22.582854 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.582790    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e"} err="failed to get container status \"7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e\": rpc error: code = NotFound desc = could not find container \"7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e\": container with ID starting with 7f5c424b9b538f5766bfb2b19c48c671435d374e95400060b77c65f687390a9e not found: ID does not exist"
Sep 17 23:55:22.582854 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.582807    2580 scope.go:117] "RemoveContainer" containerID="da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1"
Sep 17 23:55:22.583077 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:55:22.583059    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1\": container with ID starting with da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1 not found: ID does not exist" containerID="da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1"
Sep 17 23:55:22.583123 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.583083    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1"} err="failed to get container status \"da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1\": rpc error: code = NotFound desc = could not find container \"da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1\": container with ID starting with da1ea9e5b054de8308c066644876b9c66f2f6149a51d407550347e9f234dcdc1 not found: ID does not exist"
Sep 17 23:55:22.583123 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.583100    2580 scope.go:117] "RemoveContainer" containerID="a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b"
Sep 17 23:55:22.583339 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:55:22.583322    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b\": container with ID starting with a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b not found: ID does not exist" containerID="a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b"
Sep 17 23:55:22.583380 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.583344    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b"} err="failed to get container status \"a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b\": rpc error: code = NotFound desc = could not find container \"a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b\": container with ID starting with a56e7f369954c94b847d4371133e29a0018c3b9866a566a91ae9f2b530051c5b not found: ID does not exist"
Sep 17 23:55:22.583380 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.583358    2580 scope.go:117] "RemoveContainer" containerID="1b65799cb29aebebdccb7d1b6fee0de1942ccc192a81ec4baafdbc4c7fed5186"
Sep 17 23:55:22.583590 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:55:22.583576    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b65799cb29aebebdccb7d1b6fee0de1942ccc192a81ec4baafdbc4c7fed5186\": container with ID starting with 1b65799cb29aebebdccb7d1b6fee0de1942ccc192a81ec4baafdbc4c7fed5186 not found: ID does not exist" containerID="1b65799cb29aebebdccb7d1b6fee0de1942ccc192a81ec4baafdbc4c7fed5186"
Sep 17 23:55:22.583662 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:22.583593    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b65799cb29aebebdccb7d1b6fee0de1942ccc192a81ec4baafdbc4c7fed5186"} err="failed to get container status \"1b65799cb29aebebdccb7d1b6fee0de1942ccc192a81ec4baafdbc4c7fed5186\": rpc error: code = NotFound desc = could not find container \"1b65799cb29aebebdccb7d1b6fee0de1942ccc192a81ec4baafdbc4c7fed5186\": container with ID starting with 1b65799cb29aebebdccb7d1b6fee0de1942ccc192a81ec4baafdbc4c7fed5186 not found: ID does not exist"
Sep 17 23:55:23.500858 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:23.500826    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" path="/var/lib/kubelet/pods/c40cd72b-a633-4836-b616-aa0a15598e44/volumes"
Sep 17 23:55:27.519212 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:27.519172    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:55:37.519690 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:37.519651    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:55:47.519169 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:47.519133    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:55:57.519655 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:55:57.519596    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:56:07.519359 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:07.519325    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:56:13.847473 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.847436    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"]
Sep 17 23:56:13.847869 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.847710    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kube-rbac-proxy"
Sep 17 23:56:13.847869 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.847721    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kube-rbac-proxy"
Sep 17 23:56:13.847869 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.847730    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kserve-agent"
Sep 17 23:56:13.847869 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.847735    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kserve-agent"
Sep 17 23:56:13.847869 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.847745    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="storage-initializer"
Sep 17 23:56:13.847869 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.847751    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="storage-initializer"
Sep 17 23:56:13.847869 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.847757    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kserve-container"
Sep 17 23:56:13.847869 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.847762    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kserve-container"
Sep 17 23:56:13.847869 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.847807    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kserve-container"
Sep 17 23:56:13.847869 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.847814    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kube-rbac-proxy"
Sep 17 23:56:13.847869 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.847819    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40cd72b-a633-4836-b616-aa0a15598e44" containerName="kserve-agent"
Sep 17 23:56:13.852080 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.852060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:13.854148 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.854117    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:56:13.854306 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.854289    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 17 23:56:13.854437 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.854418    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 17 23:56:13.860460 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.860437    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"]
Sep 17 23:56:13.960240 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.960206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d9fbf7e-a81f-46cf-9658-0203be723a59-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:13.960240 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.960241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f24s\" (UniqueName: \"kubernetes.io/projected/3d9fbf7e-a81f-46cf-9658-0203be723a59-kube-api-access-4f24s\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:13.960448 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.960290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d9fbf7e-a81f-46cf-9658-0203be723a59-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:13.960448 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.960318    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d9fbf7e-a81f-46cf-9658-0203be723a59-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:13.960448 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:13.960362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d9fbf7e-a81f-46cf-9658-0203be723a59-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:14.061698 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:14.061654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d9fbf7e-a81f-46cf-9658-0203be723a59-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:14.061698 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:14.061703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d9fbf7e-a81f-46cf-9658-0203be723a59-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:14.061939 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:14.061734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f24s\" (UniqueName: \"kubernetes.io/projected/3d9fbf7e-a81f-46cf-9658-0203be723a59-kube-api-access-4f24s\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:14.061939 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:14.061783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d9fbf7e-a81f-46cf-9658-0203be723a59-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:14.061939 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:14.061820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d9fbf7e-a81f-46cf-9658-0203be723a59-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:14.062098 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:56:14.061981    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 17 23:56:14.062098 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:56:14.062080    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9fbf7e-a81f-46cf-9658-0203be723a59-proxy-tls podName:3d9fbf7e-a81f-46cf-9658-0203be723a59 nodeName:}" failed. No retries permitted until 2026-09-17 23:56:14.56205719 +0000 UTC m=+2707.606433347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3d9fbf7e-a81f-46cf-9658-0203be723a59-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-82wvq" (UID: "3d9fbf7e-a81f-46cf-9658-0203be723a59") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 17 23:56:14.062215 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:14.062100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d9fbf7e-a81f-46cf-9658-0203be723a59-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:14.062437 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:14.062417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d9fbf7e-a81f-46cf-9658-0203be723a59-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:14.062494 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:14.062419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d9fbf7e-a81f-46cf-9658-0203be723a59-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:14.070542 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:14.070517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f24s\" (UniqueName: \"kubernetes.io/projected/3d9fbf7e-a81f-46cf-9658-0203be723a59-kube-api-access-4f24s\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:14.563769 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:14.563733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d9fbf7e-a81f-46cf-9658-0203be723a59-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:14.563956 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:56:14.563886    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 17 23:56:14.564033 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:56:14.563957    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d9fbf7e-a81f-46cf-9658-0203be723a59-proxy-tls podName:3d9fbf7e-a81f-46cf-9658-0203be723a59 nodeName:}" failed. No retries permitted until 2026-09-17 23:56:15.563941606 +0000 UTC m=+2708.608317748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3d9fbf7e-a81f-46cf-9658-0203be723a59-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-82wvq" (UID: "3d9fbf7e-a81f-46cf-9658-0203be723a59") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 17 23:56:15.572522 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:15.572485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d9fbf7e-a81f-46cf-9658-0203be723a59-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:15.574940 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:15.574907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d9fbf7e-a81f-46cf-9658-0203be723a59-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-82wvq\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:15.663070 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:15.663036    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:15.938180 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:15.938086    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"]
Sep 17 23:56:15.940964 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:56:15.940935    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9fbf7e_a81f_46cf_9658_0203be723a59.slice/crio-b32d9692ecfbf7a88c1362e87a77ad34aa9f3a5dbf5f26896b3b9d282a8a8622 WatchSource:0}: Error finding container b32d9692ecfbf7a88c1362e87a77ad34aa9f3a5dbf5f26896b3b9d282a8a8622: Status 404 returned error can't find the container with id b32d9692ecfbf7a88c1362e87a77ad34aa9f3a5dbf5f26896b3b9d282a8a8622
Sep 17 23:56:16.753019 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:16.752983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" event={"ID":"3d9fbf7e-a81f-46cf-9658-0203be723a59","Type":"ContainerStarted","Data":"b516f35b91e89e9acb0c37c15fe4449fd326f750862d1a8b0743d0e7ab3a1bd7"}
Sep 17 23:56:16.753400 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:16.753040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" event={"ID":"3d9fbf7e-a81f-46cf-9658-0203be723a59","Type":"ContainerStarted","Data":"b32d9692ecfbf7a88c1362e87a77ad34aa9f3a5dbf5f26896b3b9d282a8a8622"}
Sep 17 23:56:17.519278 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:17.519240    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:56:20.768455 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:20.768365    2580 generic.go:356] "Generic (PLEG): container finished" podID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerID="b516f35b91e89e9acb0c37c15fe4449fd326f750862d1a8b0743d0e7ab3a1bd7" exitCode=0
Sep 17 23:56:20.768455 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:20.768411    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" event={"ID":"3d9fbf7e-a81f-46cf-9658-0203be723a59","Type":"ContainerDied","Data":"b516f35b91e89e9acb0c37c15fe4449fd326f750862d1a8b0743d0e7ab3a1bd7"}
Sep 17 23:56:20.769761 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:20.769745    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:56:27.520704 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:27.520678    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:56:28.443457 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:28.443422    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"]
Sep 17 23:56:28.444085 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:28.443820    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container" containerID="cri-o://0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d" gracePeriod=30
Sep 17 23:56:28.444085 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:28.443895    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kube-rbac-proxy" containerID="cri-o://cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826" gracePeriod=30
Sep 17 23:56:32.515029 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:32.514930    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 17 23:56:32.820339 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:32.820298    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" event={"ID":"3d9fbf7e-a81f-46cf-9658-0203be723a59","Type":"ContainerStarted","Data":"67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65"}
Sep 17 23:56:32.820525 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:32.820420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" event={"ID":"3d9fbf7e-a81f-46cf-9658-0203be723a59","Type":"ContainerStarted","Data":"7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd"}
Sep 17 23:56:32.820679 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:32.820660    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:32.822389 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:32.822367    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerID="cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826" exitCode=2
Sep 17 23:56:32.822494 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:32.822402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" event={"ID":"7e3f7233-4a4d-4274-8bf8-919e9f59608a","Type":"ContainerDied","Data":"cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826"}
Sep 17 23:56:32.840984 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:32.840943    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" podStartSLOduration=4.190628941 podStartE2EDuration="19.840929764s" totalImagesPullingTime="11.355227596s" totalInitContainerRuntime="4.295073227s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:56:32.839998025 +0000 UTC m=+2725.884374186" watchObservedRunningTime="2026-09-17 23:56:32.840929764 +0000 UTC m=+2725.885305924"
Sep 17 23:56:33.691533 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.691502    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:56:33.723212 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.723141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e3f7233-4a4d-4274-8bf8-919e9f59608a-cabundle-cert\") pod \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") "
Sep 17 23:56:33.723212 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.723183    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vnmq\" (UniqueName: \"kubernetes.io/projected/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kube-api-access-5vnmq\") pod \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") "
Sep 17 23:56:33.723212 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.723210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kube-rbac-proxy-sar-config\") pod \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") "
Sep 17 23:56:33.723541 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.723252    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e3f7233-4a4d-4274-8bf8-919e9f59608a-proxy-tls\") pod \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") "
Sep 17 23:56:33.723541 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.723270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kserve-provision-location\") pod \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\" (UID: \"7e3f7233-4a4d-4274-8bf8-919e9f59608a\") "
Sep 17 23:56:33.723690 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.723591    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3f7233-4a4d-4274-8bf8-919e9f59608a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e3f7233-4a4d-4274-8bf8-919e9f59608a" (UID: "7e3f7233-4a4d-4274-8bf8-919e9f59608a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:56:33.723690 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.723631    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e3f7233-4a4d-4274-8bf8-919e9f59608a" (UID: "7e3f7233-4a4d-4274-8bf8-919e9f59608a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:56:33.723690 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.723665    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e3f7233-4a4d-4274-8bf8-919e9f59608a" (UID: "7e3f7233-4a4d-4274-8bf8-919e9f59608a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:56:33.725499 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.725473    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kube-api-access-5vnmq" (OuterVolumeSpecName: "kube-api-access-5vnmq") pod "7e3f7233-4a4d-4274-8bf8-919e9f59608a" (UID: "7e3f7233-4a4d-4274-8bf8-919e9f59608a"). InnerVolumeSpecName "kube-api-access-5vnmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:56:33.725689 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.725670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3f7233-4a4d-4274-8bf8-919e9f59608a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e3f7233-4a4d-4274-8bf8-919e9f59608a" (UID: "7e3f7233-4a4d-4274-8bf8-919e9f59608a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:56:33.823823 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.823789    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e3f7233-4a4d-4274-8bf8-919e9f59608a-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:56:33.823823 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.823819    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:56:33.823823 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.823830    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e3f7233-4a4d-4274-8bf8-919e9f59608a-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:56:33.824083 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.823840    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vnmq\" (UniqueName: \"kubernetes.io/projected/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kube-api-access-5vnmq\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:56:33.824083 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.823850    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e3f7233-4a4d-4274-8bf8-919e9f59608a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:56:33.828333 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.828305    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerID="0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d" exitCode=0
Sep 17 23:56:33.828395 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.828369    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"
Sep 17 23:56:33.828434 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.828386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" event={"ID":"7e3f7233-4a4d-4274-8bf8-919e9f59608a","Type":"ContainerDied","Data":"0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d"}
Sep 17 23:56:33.828468 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.828449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd" event={"ID":"7e3f7233-4a4d-4274-8bf8-919e9f59608a","Type":"ContainerDied","Data":"0bc6c115edcb53ce663a0b213f0ddb4bd2c49444d8b817727022563fb0bebc9a"}
Sep 17 23:56:33.828505 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.828476    2580 scope.go:117] "RemoveContainer" containerID="cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826"
Sep 17 23:56:33.828970 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.828943    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:33.830658 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.830606    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:56:33.837309 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.837293    2580 scope.go:117] "RemoveContainer" containerID="0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d"
Sep 17 23:56:33.844526 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.844504    2580 scope.go:117] "RemoveContainer" containerID="ed3750dc49303bfbf4c778ffb90c8bda588da1ca6424c65597b9861edb1ecaaf"
Sep 17 23:56:33.849940 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.849919    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"]
Sep 17 23:56:33.852161 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.852139    2580 scope.go:117] "RemoveContainer" containerID="cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826"
Sep 17 23:56:33.852418 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:56:33.852402    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826\": container with ID starting with cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826 not found: ID does not exist" containerID="cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826"
Sep 17 23:56:33.852487 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.852430    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826"} err="failed to get container status \"cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826\": rpc error: code = NotFound desc = could not find container \"cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826\": container with ID starting with cd422fb47c9fb29c9d6a9e0cd43cb13772dc618595816defcd48225f3415b826 not found: ID does not exist"
Sep 17 23:56:33.852487 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.852450    2580 scope.go:117] "RemoveContainer" containerID="0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d"
Sep 17 23:56:33.852716 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:56:33.852687    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d\": container with ID starting with 0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d not found: ID does not exist" containerID="0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d"
Sep 17 23:56:33.852795 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.852726    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d"} err="failed to get container status \"0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d\": rpc error: code = NotFound desc = could not find container \"0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d\": container with ID starting with 0fa66be5e29a2f094f11714abeb0d0d3e3f30bc0988c27254b91f8a9ae460d0d not found: ID does not exist"
Sep 17 23:56:33.852795 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.852748    2580 scope.go:117] "RemoveContainer" containerID="ed3750dc49303bfbf4c778ffb90c8bda588da1ca6424c65597b9861edb1ecaaf"
Sep 17 23:56:33.853072 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:56:33.853046    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3750dc49303bfbf4c778ffb90c8bda588da1ca6424c65597b9861edb1ecaaf\": container with ID starting with ed3750dc49303bfbf4c778ffb90c8bda588da1ca6424c65597b9861edb1ecaaf not found: ID does not exist" containerID="ed3750dc49303bfbf4c778ffb90c8bda588da1ca6424c65597b9861edb1ecaaf"
Sep 17 23:56:33.853122 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.853080    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3750dc49303bfbf4c778ffb90c8bda588da1ca6424c65597b9861edb1ecaaf"} err="failed to get container status \"ed3750dc49303bfbf4c778ffb90c8bda588da1ca6424c65597b9861edb1ecaaf\": rpc error: code = NotFound desc = could not find container \"ed3750dc49303bfbf4c778ffb90c8bda588da1ca6424c65597b9861edb1ecaaf\": container with ID starting with ed3750dc49303bfbf4c778ffb90c8bda588da1ca6424c65597b9861edb1ecaaf not found: ID does not exist"
Sep 17 23:56:33.855874 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:33.855854    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-98f74b798-7qxkd"]
Sep 17 23:56:34.833225 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:34.833194    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:56:35.500984 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:35.500943    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" path="/var/lib/kubelet/pods/7e3f7233-4a4d-4274-8bf8-919e9f59608a/volumes"
Sep 17 23:56:39.838324 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:39.838299    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:56:39.838810 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:39.838783    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:56:49.839387 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:49.839349    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:56:59.839712 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:56:59.839673    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:57:09.839685 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:09.839641    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:57:19.839636 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:19.839587    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:57:25.192632 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:25.192590    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"]
Sep 17 23:57:25.193005 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:25.192870    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kserve-container" containerID="cri-o://7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd" gracePeriod=30
Sep 17 23:57:25.193005 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:25.192886    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kube-rbac-proxy" containerID="cri-o://67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65" gracePeriod=30
Sep 17 23:57:26.012877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:26.012842    2580 generic.go:356] "Generic (PLEG): container finished" podID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerID="67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65" exitCode=2
Sep 17 23:57:26.013048 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:26.012920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" event={"ID":"3d9fbf7e-a81f-46cf-9658-0203be723a59","Type":"ContainerDied","Data":"67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65"}
Sep 17 23:57:28.227550 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.227523    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:57:28.316877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.316839    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f24s\" (UniqueName: \"kubernetes.io/projected/3d9fbf7e-a81f-46cf-9658-0203be723a59-kube-api-access-4f24s\") pod \"3d9fbf7e-a81f-46cf-9658-0203be723a59\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") "
Sep 17 23:57:28.316877 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.316880    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d9fbf7e-a81f-46cf-9658-0203be723a59-kserve-provision-location\") pod \"3d9fbf7e-a81f-46cf-9658-0203be723a59\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") "
Sep 17 23:57:28.317134 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.316927    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d9fbf7e-a81f-46cf-9658-0203be723a59-kube-rbac-proxy-sar-config\") pod \"3d9fbf7e-a81f-46cf-9658-0203be723a59\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") "
Sep 17 23:57:28.317134 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.316952    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d9fbf7e-a81f-46cf-9658-0203be723a59-cabundle-cert\") pod \"3d9fbf7e-a81f-46cf-9658-0203be723a59\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") "
Sep 17 23:57:28.317134 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.316991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d9fbf7e-a81f-46cf-9658-0203be723a59-proxy-tls\") pod \"3d9fbf7e-a81f-46cf-9658-0203be723a59\" (UID: \"3d9fbf7e-a81f-46cf-9658-0203be723a59\") "
Sep 17 23:57:28.317377 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.317351    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9fbf7e-a81f-46cf-9658-0203be723a59-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d9fbf7e-a81f-46cf-9658-0203be723a59" (UID: "3d9fbf7e-a81f-46cf-9658-0203be723a59"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:57:28.317443 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.317396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9fbf7e-a81f-46cf-9658-0203be723a59-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d9fbf7e-a81f-46cf-9658-0203be723a59" (UID: "3d9fbf7e-a81f-46cf-9658-0203be723a59"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:57:28.319238 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.319161    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9fbf7e-a81f-46cf-9658-0203be723a59-kube-api-access-4f24s" (OuterVolumeSpecName: "kube-api-access-4f24s") pod "3d9fbf7e-a81f-46cf-9658-0203be723a59" (UID: "3d9fbf7e-a81f-46cf-9658-0203be723a59"). InnerVolumeSpecName "kube-api-access-4f24s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:57:28.319357 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.319317    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d9fbf7e-a81f-46cf-9658-0203be723a59-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d9fbf7e-a81f-46cf-9658-0203be723a59" (UID: "3d9fbf7e-a81f-46cf-9658-0203be723a59"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:57:28.325455 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.325418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9fbf7e-a81f-46cf-9658-0203be723a59-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d9fbf7e-a81f-46cf-9658-0203be723a59" (UID: "3d9fbf7e-a81f-46cf-9658-0203be723a59"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:57:28.418423 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.418392    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f24s\" (UniqueName: \"kubernetes.io/projected/3d9fbf7e-a81f-46cf-9658-0203be723a59-kube-api-access-4f24s\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:57:28.418423 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.418419    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d9fbf7e-a81f-46cf-9658-0203be723a59-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:57:28.418423 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.418428    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d9fbf7e-a81f-46cf-9658-0203be723a59-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:57:28.418658 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.418438    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d9fbf7e-a81f-46cf-9658-0203be723a59-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:57:28.418658 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:28.418447    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d9fbf7e-a81f-46cf-9658-0203be723a59-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:57:29.024343 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.024306    2580 generic.go:356] "Generic (PLEG): container finished" podID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerID="7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd" exitCode=0
Sep 17 23:57:29.024522 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.024366    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"
Sep 17 23:57:29.024522 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.024387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" event={"ID":"3d9fbf7e-a81f-46cf-9658-0203be723a59","Type":"ContainerDied","Data":"7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd"}
Sep 17 23:57:29.024522 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.024449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq" event={"ID":"3d9fbf7e-a81f-46cf-9658-0203be723a59","Type":"ContainerDied","Data":"b32d9692ecfbf7a88c1362e87a77ad34aa9f3a5dbf5f26896b3b9d282a8a8622"}
Sep 17 23:57:29.024522 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.024475    2580 scope.go:117] "RemoveContainer" containerID="67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65"
Sep 17 23:57:29.032990 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.032966    2580 scope.go:117] "RemoveContainer" containerID="7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd"
Sep 17 23:57:29.040123 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.040102    2580 scope.go:117] "RemoveContainer" containerID="b516f35b91e89e9acb0c37c15fe4449fd326f750862d1a8b0743d0e7ab3a1bd7"
Sep 17 23:57:29.047680 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.047627    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"]
Sep 17 23:57:29.047749 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.047719    2580 scope.go:117] "RemoveContainer" containerID="67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65"
Sep 17 23:57:29.047995 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:57:29.047970    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65\": container with ID starting with 67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65 not found: ID does not exist" containerID="67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65"
Sep 17 23:57:29.048044 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.048001    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65"} err="failed to get container status \"67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65\": rpc error: code = NotFound desc = could not find container \"67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65\": container with ID starting with 67b828e38fb07ec67c357dfe390385d37e2f234fac0d4b32ad27ff8cb3b04c65 not found: ID does not exist"
Sep 17 23:57:29.048044 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.048017    2580 scope.go:117] "RemoveContainer" containerID="7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd"
Sep 17 23:57:29.048260 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:57:29.048245    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd\": container with ID starting with 7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd not found: ID does not exist" containerID="7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd"
Sep 17 23:57:29.048309 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.048262    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd"} err="failed to get container status \"7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd\": rpc error: code = NotFound desc = could not find container \"7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd\": container with ID starting with 7479059368c4f1da3558fe0ac35facf397cf1f79a7bfee5130493220f31ddedd not found: ID does not exist"
Sep 17 23:57:29.048309 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.048273    2580 scope.go:117] "RemoveContainer" containerID="b516f35b91e89e9acb0c37c15fe4449fd326f750862d1a8b0743d0e7ab3a1bd7"
Sep 17 23:57:29.048488 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:57:29.048470    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b516f35b91e89e9acb0c37c15fe4449fd326f750862d1a8b0743d0e7ab3a1bd7\": container with ID starting with b516f35b91e89e9acb0c37c15fe4449fd326f750862d1a8b0743d0e7ab3a1bd7 not found: ID does not exist" containerID="b516f35b91e89e9acb0c37c15fe4449fd326f750862d1a8b0743d0e7ab3a1bd7"
Sep 17 23:57:29.048528 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.048495    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b516f35b91e89e9acb0c37c15fe4449fd326f750862d1a8b0743d0e7ab3a1bd7"} err="failed to get container status \"b516f35b91e89e9acb0c37c15fe4449fd326f750862d1a8b0743d0e7ab3a1bd7\": rpc error: code = NotFound desc = could not find container \"b516f35b91e89e9acb0c37c15fe4449fd326f750862d1a8b0743d0e7ab3a1bd7\": container with ID starting with b516f35b91e89e9acb0c37c15fe4449fd326f750862d1a8b0743d0e7ab3a1bd7 not found: ID does not exist"
Sep 17 23:57:29.051341 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.051313    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-82wvq"]
Sep 17 23:57:29.500166 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:29.500132    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" path="/var/lib/kubelet/pods/3d9fbf7e-a81f-46cf-9658-0203be723a59/volumes"
Sep 17 23:57:30.381597 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381562    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"]
Sep 17 23:57:30.381831 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381818    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="storage-initializer"
Sep 17 23:57:30.381874 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381833    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="storage-initializer"
Sep 17 23:57:30.381874 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381842    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="storage-initializer"
Sep 17 23:57:30.381874 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381847    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="storage-initializer"
Sep 17 23:57:30.381874 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381871    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kube-rbac-proxy"
Sep 17 23:57:30.382015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381878    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kube-rbac-proxy"
Sep 17 23:57:30.382015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381883    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kserve-container"
Sep 17 23:57:30.382015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381889    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kserve-container"
Sep 17 23:57:30.382015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381897    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kube-rbac-proxy"
Sep 17 23:57:30.382015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381902    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kube-rbac-proxy"
Sep 17 23:57:30.382015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381909    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container"
Sep 17 23:57:30.382015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381913    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container"
Sep 17 23:57:30.382015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381962    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kserve-container"
Sep 17 23:57:30.382015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381969    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kube-rbac-proxy"
Sep 17 23:57:30.382015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381974    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3f7233-4a4d-4274-8bf8-919e9f59608a" containerName="kube-rbac-proxy"
Sep 17 23:57:30.382015 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.381981    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9fbf7e-a81f-46cf-9658-0203be723a59" containerName="kserve-container"
Sep 17 23:57:30.386388 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.386367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.388806 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.388760    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:57:30.388806 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.388788    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:57:30.388806 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.388797    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 23:57:30.389787 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.389766    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:57:30.389885 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.389807    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:57:30.389885 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.389819    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 17 23:57:30.389885 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.389813    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:57:30.394159 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.394133    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"]
Sep 17 23:57:30.533792 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.533753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/829b5680-62dc-4a50-bfdc-ba719b1279c0-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.533792 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.533797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcspk\" (UniqueName: \"kubernetes.io/projected/829b5680-62dc-4a50-bfdc-ba719b1279c0-kube-api-access-mcspk\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.534283 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.533825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/829b5680-62dc-4a50-bfdc-ba719b1279c0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.534283 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.533871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/829b5680-62dc-4a50-bfdc-ba719b1279c0-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.534283 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.533939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/829b5680-62dc-4a50-bfdc-ba719b1279c0-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.634545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.634461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/829b5680-62dc-4a50-bfdc-ba719b1279c0-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.634545 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.634514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/829b5680-62dc-4a50-bfdc-ba719b1279c0-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.634867 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.634546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcspk\" (UniqueName: \"kubernetes.io/projected/829b5680-62dc-4a50-bfdc-ba719b1279c0-kube-api-access-mcspk\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.634867 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.634573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/829b5680-62dc-4a50-bfdc-ba719b1279c0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.634867 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.634641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/829b5680-62dc-4a50-bfdc-ba719b1279c0-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.634867 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:57:30.634749    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 17 23:57:30.634867 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:57:30.634832    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/829b5680-62dc-4a50-bfdc-ba719b1279c0-proxy-tls podName:829b5680-62dc-4a50-bfdc-ba719b1279c0 nodeName:}" failed. No retries permitted until 2026-09-17 23:57:31.134809364 +0000 UTC m=+2784.179185502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/829b5680-62dc-4a50-bfdc-ba719b1279c0-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" (UID: "829b5680-62dc-4a50-bfdc-ba719b1279c0") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 17 23:57:30.635115 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.634967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/829b5680-62dc-4a50-bfdc-ba719b1279c0-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.635256 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.635239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/829b5680-62dc-4a50-bfdc-ba719b1279c0-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.635289 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.635243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/829b5680-62dc-4a50-bfdc-ba719b1279c0-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:30.643577 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:30.643552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcspk\" (UniqueName: \"kubernetes.io/projected/829b5680-62dc-4a50-bfdc-ba719b1279c0-kube-api-access-mcspk\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:31.138481 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:31.138440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/829b5680-62dc-4a50-bfdc-ba719b1279c0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:31.141022 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:31.140996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/829b5680-62dc-4a50-bfdc-ba719b1279c0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:31.297912 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:31.297881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:31.571137 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:31.571105    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"]
Sep 17 23:57:31.574516 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:57:31.574491    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829b5680_62dc_4a50_bfdc_ba719b1279c0.slice/crio-01d97dde7050cb8aadff3feca243dd2b6b752c32e667d2a25ade1d9d1f3e3228 WatchSource:0}: Error finding container 01d97dde7050cb8aadff3feca243dd2b6b752c32e667d2a25ade1d9d1f3e3228: Status 404 returned error can't find the container with id 01d97dde7050cb8aadff3feca243dd2b6b752c32e667d2a25ade1d9d1f3e3228
Sep 17 23:57:32.037336 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:32.037299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" event={"ID":"829b5680-62dc-4a50-bfdc-ba719b1279c0","Type":"ContainerStarted","Data":"b719c068e882912cdaa97845aea03c6d260c29070c5e8531ca5b62a8b739865c"}
Sep 17 23:57:32.037480 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:32.037373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" event={"ID":"829b5680-62dc-4a50-bfdc-ba719b1279c0","Type":"ContainerStarted","Data":"01d97dde7050cb8aadff3feca243dd2b6b752c32e667d2a25ade1d9d1f3e3228"}
Sep 17 23:57:37.055041 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:37.055009    2580 generic.go:356] "Generic (PLEG): container finished" podID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerID="b719c068e882912cdaa97845aea03c6d260c29070c5e8531ca5b62a8b739865c" exitCode=0
Sep 17 23:57:37.055459 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:37.055045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" event={"ID":"829b5680-62dc-4a50-bfdc-ba719b1279c0","Type":"ContainerDied","Data":"b719c068e882912cdaa97845aea03c6d260c29070c5e8531ca5b62a8b739865c"}
Sep 17 23:57:38.060350 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:38.060319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" event={"ID":"829b5680-62dc-4a50-bfdc-ba719b1279c0","Type":"ContainerStarted","Data":"9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af"}
Sep 17 23:57:38.060798 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:38.060379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" event={"ID":"829b5680-62dc-4a50-bfdc-ba719b1279c0","Type":"ContainerStarted","Data":"61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9"}
Sep 17 23:57:38.060798 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:38.060650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:38.060898 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:38.060801    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:38.061990 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:38.061966    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 23:57:38.078892 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:38.078836    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" podStartSLOduration=3.3588474 podStartE2EDuration="8.07881836s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.71997096s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:57:38.077659538 +0000 UTC m=+2791.122035700" watchObservedRunningTime="2026-09-17 23:57:38.07881836 +0000 UTC m=+2791.123194521"
Sep 17 23:57:39.064596 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:39.064553    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 23:57:44.069513 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:44.069481    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:57:44.070032 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:44.070002    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 23:57:54.070741 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:57:54.070702    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 23:58:04.070786 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:04.070745    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 23:58:14.070832 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:14.070792    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 23:58:24.070494 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:24.070462    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:58:31.999307 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:31.999270    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"]
Sep 17 23:58:31.999816 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:31.999509    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kserve-container" containerID="cri-o://61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9" gracePeriod=30
Sep 17 23:58:31.999816 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:31.999550    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kube-rbac-proxy" containerID="cri-o://9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af" gracePeriod=30
Sep 17 23:58:32.258597 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:32.258519    2580 generic.go:356] "Generic (PLEG): container finished" podID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerID="9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af" exitCode=2
Sep 17 23:58:32.258782 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:32.258590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" event={"ID":"829b5680-62dc-4a50-bfdc-ba719b1279c0","Type":"ContainerDied","Data":"9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af"}
Sep 17 23:58:34.065568 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:34.065525    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 23:58:34.069930 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:34.069906    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 23:58:35.037995 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.037971    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:58:35.118024 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.117950    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/829b5680-62dc-4a50-bfdc-ba719b1279c0-kube-rbac-proxy-sar-config\") pod \"829b5680-62dc-4a50-bfdc-ba719b1279c0\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") "
Sep 17 23:58:35.118024 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.118013    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/829b5680-62dc-4a50-bfdc-ba719b1279c0-proxy-tls\") pod \"829b5680-62dc-4a50-bfdc-ba719b1279c0\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") "
Sep 17 23:58:35.118505 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.118036    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/829b5680-62dc-4a50-bfdc-ba719b1279c0-kserve-provision-location\") pod \"829b5680-62dc-4a50-bfdc-ba719b1279c0\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") "
Sep 17 23:58:35.118505 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.118053    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/829b5680-62dc-4a50-bfdc-ba719b1279c0-cabundle-cert\") pod \"829b5680-62dc-4a50-bfdc-ba719b1279c0\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") "
Sep 17 23:58:35.118505 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.118083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcspk\" (UniqueName: \"kubernetes.io/projected/829b5680-62dc-4a50-bfdc-ba719b1279c0-kube-api-access-mcspk\") pod \"829b5680-62dc-4a50-bfdc-ba719b1279c0\" (UID: \"829b5680-62dc-4a50-bfdc-ba719b1279c0\") "
Sep 17 23:58:35.118505 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.118456    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829b5680-62dc-4a50-bfdc-ba719b1279c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "829b5680-62dc-4a50-bfdc-ba719b1279c0" (UID: "829b5680-62dc-4a50-bfdc-ba719b1279c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:58:35.118729 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.118568    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829b5680-62dc-4a50-bfdc-ba719b1279c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "829b5680-62dc-4a50-bfdc-ba719b1279c0" (UID: "829b5680-62dc-4a50-bfdc-ba719b1279c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:58:35.120309 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.120284    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829b5680-62dc-4a50-bfdc-ba719b1279c0-kube-api-access-mcspk" (OuterVolumeSpecName: "kube-api-access-mcspk") pod "829b5680-62dc-4a50-bfdc-ba719b1279c0" (UID: "829b5680-62dc-4a50-bfdc-ba719b1279c0"). InnerVolumeSpecName "kube-api-access-mcspk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:58:35.120428 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.120319    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829b5680-62dc-4a50-bfdc-ba719b1279c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "829b5680-62dc-4a50-bfdc-ba719b1279c0" (UID: "829b5680-62dc-4a50-bfdc-ba719b1279c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:58:35.129492 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.129466    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829b5680-62dc-4a50-bfdc-ba719b1279c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "829b5680-62dc-4a50-bfdc-ba719b1279c0" (UID: "829b5680-62dc-4a50-bfdc-ba719b1279c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:58:35.219034 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.219006    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/829b5680-62dc-4a50-bfdc-ba719b1279c0-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:58:35.219034 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.219029    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcspk\" (UniqueName: \"kubernetes.io/projected/829b5680-62dc-4a50-bfdc-ba719b1279c0-kube-api-access-mcspk\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:58:35.219034 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.219040    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/829b5680-62dc-4a50-bfdc-ba719b1279c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:58:35.219218 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.219048    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/829b5680-62dc-4a50-bfdc-ba719b1279c0-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:58:35.219218 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.219056    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/829b5680-62dc-4a50-bfdc-ba719b1279c0-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:58:35.272562 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.272527    2580 generic.go:356] "Generic (PLEG): container finished" podID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerID="61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9" exitCode=0
Sep 17 23:58:35.272665 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.272591    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"
Sep 17 23:58:35.272665 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.272603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" event={"ID":"829b5680-62dc-4a50-bfdc-ba719b1279c0","Type":"ContainerDied","Data":"61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9"}
Sep 17 23:58:35.272752 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.272690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck" event={"ID":"829b5680-62dc-4a50-bfdc-ba719b1279c0","Type":"ContainerDied","Data":"01d97dde7050cb8aadff3feca243dd2b6b752c32e667d2a25ade1d9d1f3e3228"}
Sep 17 23:58:35.272752 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.272720    2580 scope.go:117] "RemoveContainer" containerID="9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af"
Sep 17 23:58:35.280770 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.280752    2580 scope.go:117] "RemoveContainer" containerID="61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9"
Sep 17 23:58:35.287740 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.287726    2580 scope.go:117] "RemoveContainer" containerID="b719c068e882912cdaa97845aea03c6d260c29070c5e8531ca5b62a8b739865c"
Sep 17 23:58:35.293392 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.293368    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"]
Sep 17 23:58:35.294784 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.294762    2580 scope.go:117] "RemoveContainer" containerID="9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af"
Sep 17 23:58:35.295057 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:58:35.295037    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af\": container with ID starting with 9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af not found: ID does not exist" containerID="9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af"
Sep 17 23:58:35.295159 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.295066    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af"} err="failed to get container status \"9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af\": rpc error: code = NotFound desc = could not find container \"9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af\": container with ID starting with 9a9458c299f90bbae36bbc421c2e37831b995441f494aef98fa11dd2ca4223af not found: ID does not exist"
Sep 17 23:58:35.295159 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.295083    2580 scope.go:117] "RemoveContainer" containerID="61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9"
Sep 17 23:58:35.295366 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:58:35.295348    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9\": container with ID starting with 61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9 not found: ID does not exist" containerID="61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9"
Sep 17 23:58:35.295410 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.295372    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9"} err="failed to get container status \"61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9\": rpc error: code = NotFound desc = could not find container \"61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9\": container with ID starting with 61cc8327cb9157a61e9226ed28e2bffd293929810f57df629ac3ce0055ad90b9 not found: ID does not exist"
Sep 17 23:58:35.295410 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.295388    2580 scope.go:117] "RemoveContainer" containerID="b719c068e882912cdaa97845aea03c6d260c29070c5e8531ca5b62a8b739865c"
Sep 17 23:58:35.295664 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:58:35.295644    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b719c068e882912cdaa97845aea03c6d260c29070c5e8531ca5b62a8b739865c\": container with ID starting with b719c068e882912cdaa97845aea03c6d260c29070c5e8531ca5b62a8b739865c not found: ID does not exist" containerID="b719c068e882912cdaa97845aea03c6d260c29070c5e8531ca5b62a8b739865c"
Sep 17 23:58:35.295759 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.295665    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b719c068e882912cdaa97845aea03c6d260c29070c5e8531ca5b62a8b739865c"} err="failed to get container status \"b719c068e882912cdaa97845aea03c6d260c29070c5e8531ca5b62a8b739865c\": rpc error: code = NotFound desc = could not find container \"b719c068e882912cdaa97845aea03c6d260c29070c5e8531ca5b62a8b739865c\": container with ID starting with b719c068e882912cdaa97845aea03c6d260c29070c5e8531ca5b62a8b739865c not found: ID does not exist"
Sep 17 23:58:35.297410 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.297390    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d2gck"]
Sep 17 23:58:35.500896 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:58:35.500817    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" path="/var/lib/kubelet/pods/829b5680-62dc-4a50-bfdc-ba719b1279c0/volumes"
Sep 17 23:59:33.988976 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.988944    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"]
Sep 17 23:59:33.989424 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.989177    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="storage-initializer"
Sep 17 23:59:33.989424 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.989188    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="storage-initializer"
Sep 17 23:59:33.989424 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.989197    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kserve-container"
Sep 17 23:59:33.989424 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.989202    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kserve-container"
Sep 17 23:59:33.989424 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.989215    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kube-rbac-proxy"
Sep 17 23:59:33.989424 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.989226    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kube-rbac-proxy"
Sep 17 23:59:33.989424 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.989279    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kube-rbac-proxy"
Sep 17 23:59:33.989424 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.989287    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="829b5680-62dc-4a50-bfdc-ba719b1279c0" containerName="kserve-container"
Sep 17 23:59:33.992194 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.992177    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:33.994301 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.994278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 17 23:59:33.994531 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.994512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 17 23:59:33.994531 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.994523    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:59:33.994667 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.994626    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 23:59:33.995305 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.995286    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:59:33.995391 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:33.995321    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:59:34.002749 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.002728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"]
Sep 17 23:59:34.037742 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.037715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0320fd0a-b61a-42a5-97a6-52296a7cb940-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.037742 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.037749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0320fd0a-b61a-42a5-97a6-52296a7cb940-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.037946 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.037772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0320fd0a-b61a-42a5-97a6-52296a7cb940-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.037946 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.037830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wcv2\" (UniqueName: \"kubernetes.io/projected/0320fd0a-b61a-42a5-97a6-52296a7cb940-kube-api-access-9wcv2\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.138780 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.138744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0320fd0a-b61a-42a5-97a6-52296a7cb940-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.139021 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.138999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0320fd0a-b61a-42a5-97a6-52296a7cb940-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.139151 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.139136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0320fd0a-b61a-42a5-97a6-52296a7cb940-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.139311 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.139297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wcv2\" (UniqueName: \"kubernetes.io/projected/0320fd0a-b61a-42a5-97a6-52296a7cb940-kube-api-access-9wcv2\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.140554 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.140521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0320fd0a-b61a-42a5-97a6-52296a7cb940-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.141745 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.141598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0320fd0a-b61a-42a5-97a6-52296a7cb940-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.144775 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.144746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0320fd0a-b61a-42a5-97a6-52296a7cb940-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.147919 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.147893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wcv2\" (UniqueName: \"kubernetes.io/projected/0320fd0a-b61a-42a5-97a6-52296a7cb940-kube-api-access-9wcv2\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.302739 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.302707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:34.572936 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:34.572838    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"]
Sep 17 23:59:34.576576 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:59:34.576548    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0320fd0a_b61a_42a5_97a6_52296a7cb940.slice/crio-3f9a37bf5cb7736e9d9a999fa2a6361c2320f0baee103df41ed93867e4f77887 WatchSource:0}: Error finding container 3f9a37bf5cb7736e9d9a999fa2a6361c2320f0baee103df41ed93867e4f77887: Status 404 returned error can't find the container with id 3f9a37bf5cb7736e9d9a999fa2a6361c2320f0baee103df41ed93867e4f77887
Sep 17 23:59:35.481508 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:35.481474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw" event={"ID":"0320fd0a-b61a-42a5-97a6-52296a7cb940","Type":"ContainerStarted","Data":"ced41f7a0599a760bf0eccbdd348756ae96571f3a40e8074134702ec854301c3"}
Sep 17 23:59:35.481895 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:35.481549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw" event={"ID":"0320fd0a-b61a-42a5-97a6-52296a7cb940","Type":"ContainerStarted","Data":"3f9a37bf5cb7736e9d9a999fa2a6361c2320f0baee103df41ed93867e4f77887"}
Sep 17 23:59:41.502952 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:41.502926    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw_0320fd0a-b61a-42a5-97a6-52296a7cb940/storage-initializer/0.log"
Sep 17 23:59:41.503460 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:41.502963    2580 generic.go:356] "Generic (PLEG): container finished" podID="0320fd0a-b61a-42a5-97a6-52296a7cb940" containerID="ced41f7a0599a760bf0eccbdd348756ae96571f3a40e8074134702ec854301c3" exitCode=1
Sep 17 23:59:41.503460 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:41.503049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw" event={"ID":"0320fd0a-b61a-42a5-97a6-52296a7cb940","Type":"ContainerDied","Data":"ced41f7a0599a760bf0eccbdd348756ae96571f3a40e8074134702ec854301c3"}
Sep 17 23:59:42.508682 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:42.508650    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw_0320fd0a-b61a-42a5-97a6-52296a7cb940/storage-initializer/0.log"
Sep 17 23:59:42.509083 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:42.508729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw" event={"ID":"0320fd0a-b61a-42a5-97a6-52296a7cb940","Type":"ContainerStarted","Data":"b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5"}
Sep 17 23:59:43.967308 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:43.967277    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"]
Sep 17 23:59:43.967769 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:43.967484    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw" podUID="0320fd0a-b61a-42a5-97a6-52296a7cb940" containerName="storage-initializer" containerID="cri-o://b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5" gracePeriod=30
Sep 17 23:59:45.127596 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.127559    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"]
Sep 17 23:59:45.130916 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.130900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.133001 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.132975    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:59:45.133124 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.133056    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 17 23:59:45.133277 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.133263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:59:45.133337 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.133296    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 23:59:45.139936 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.139916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"]
Sep 17 23:59:45.212387 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.212354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.212387 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.212390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.212652 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.212410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.212652 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.212444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxj45\" (UniqueName: \"kubernetes.io/projected/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kube-api-access-zxj45\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.212652 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.212460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.313271 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.313237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.313271 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.313274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.313464 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.313302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.313464 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.313335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxj45\" (UniqueName: \"kubernetes.io/projected/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kube-api-access-zxj45\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.313464 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.313356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.313779 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.313757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.314147 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.314128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.314188 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.314128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.315922 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.315905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.321917 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.321899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxj45\" (UniqueName: \"kubernetes.io/projected/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kube-api-access-zxj45\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.440936 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.440853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:45.712182 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.712105    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"]
Sep 17 23:59:45.715393 ip-10-0-128-184 kubenswrapper[2580]: W0917 23:59:45.715351    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbfa1b6_b513_4ad1_9b11_924d6e71bfbd.slice/crio-1998b72e667a29499bfc2288998256d5e861fc07cd7804889ea58c6c19846f8f WatchSource:0}: Error finding container 1998b72e667a29499bfc2288998256d5e861fc07cd7804889ea58c6c19846f8f: Status 404 returned error can't find the container with id 1998b72e667a29499bfc2288998256d5e861fc07cd7804889ea58c6c19846f8f
Sep 17 23:59:45.923384 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.923360    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw_0320fd0a-b61a-42a5-97a6-52296a7cb940/storage-initializer/1.log"
Sep 17 23:59:45.923804 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.923787    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw_0320fd0a-b61a-42a5-97a6-52296a7cb940/storage-initializer/0.log"
Sep 17 23:59:45.926114 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:45.926099    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:46.018965 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.018874    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0320fd0a-b61a-42a5-97a6-52296a7cb940-proxy-tls\") pod \"0320fd0a-b61a-42a5-97a6-52296a7cb940\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") "
Sep 17 23:59:46.018965 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.018945    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0320fd0a-b61a-42a5-97a6-52296a7cb940-kserve-provision-location\") pod \"0320fd0a-b61a-42a5-97a6-52296a7cb940\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") "
Sep 17 23:59:46.019211 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.018990    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wcv2\" (UniqueName: \"kubernetes.io/projected/0320fd0a-b61a-42a5-97a6-52296a7cb940-kube-api-access-9wcv2\") pod \"0320fd0a-b61a-42a5-97a6-52296a7cb940\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") "
Sep 17 23:59:46.019211 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.019032    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0320fd0a-b61a-42a5-97a6-52296a7cb940-kube-rbac-proxy-sar-config\") pod \"0320fd0a-b61a-42a5-97a6-52296a7cb940\" (UID: \"0320fd0a-b61a-42a5-97a6-52296a7cb940\") "
Sep 17 23:59:46.019324 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.019213    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0320fd0a-b61a-42a5-97a6-52296a7cb940-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0320fd0a-b61a-42a5-97a6-52296a7cb940" (UID: "0320fd0a-b61a-42a5-97a6-52296a7cb940"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:59:46.019462 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.019438    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0320fd0a-b61a-42a5-97a6-52296a7cb940-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0320fd0a-b61a-42a5-97a6-52296a7cb940" (UID: "0320fd0a-b61a-42a5-97a6-52296a7cb940"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:59:46.021136 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.021111    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0320fd0a-b61a-42a5-97a6-52296a7cb940-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0320fd0a-b61a-42a5-97a6-52296a7cb940" (UID: "0320fd0a-b61a-42a5-97a6-52296a7cb940"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:59:46.021136 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.021109    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0320fd0a-b61a-42a5-97a6-52296a7cb940-kube-api-access-9wcv2" (OuterVolumeSpecName: "kube-api-access-9wcv2") pod "0320fd0a-b61a-42a5-97a6-52296a7cb940" (UID: "0320fd0a-b61a-42a5-97a6-52296a7cb940"). InnerVolumeSpecName "kube-api-access-9wcv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:59:46.119761 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.119726    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0320fd0a-b61a-42a5-97a6-52296a7cb940-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:59:46.119761 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.119752    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wcv2\" (UniqueName: \"kubernetes.io/projected/0320fd0a-b61a-42a5-97a6-52296a7cb940-kube-api-access-9wcv2\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:59:46.119761 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.119762    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0320fd0a-b61a-42a5-97a6-52296a7cb940-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:59:46.119761 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.119771    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0320fd0a-b61a-42a5-97a6-52296a7cb940-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 17 23:59:46.523895 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.523854    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" event={"ID":"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd","Type":"ContainerStarted","Data":"73fb6d9c63eb4737ea7d3cb1143a67eca41f403f325640660f648d528c4af825"}
Sep 17 23:59:46.524346 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.523944    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" event={"ID":"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd","Type":"ContainerStarted","Data":"1998b72e667a29499bfc2288998256d5e861fc07cd7804889ea58c6c19846f8f"}
Sep 17 23:59:46.525302 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.525281    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw_0320fd0a-b61a-42a5-97a6-52296a7cb940/storage-initializer/1.log"
Sep 17 23:59:46.525711 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.525695    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw_0320fd0a-b61a-42a5-97a6-52296a7cb940/storage-initializer/0.log"
Sep 17 23:59:46.525773 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.525729    2580 generic.go:356] "Generic (PLEG): container finished" podID="0320fd0a-b61a-42a5-97a6-52296a7cb940" containerID="b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5" exitCode=1
Sep 17 23:59:46.525773 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.525757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw" event={"ID":"0320fd0a-b61a-42a5-97a6-52296a7cb940","Type":"ContainerDied","Data":"b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5"}
Sep 17 23:59:46.525855 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.525792    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"
Sep 17 23:59:46.525855 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.525805    2580 scope.go:117] "RemoveContainer" containerID="b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5"
Sep 17 23:59:46.525933 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.525795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw" event={"ID":"0320fd0a-b61a-42a5-97a6-52296a7cb940","Type":"ContainerDied","Data":"3f9a37bf5cb7736e9d9a999fa2a6361c2320f0baee103df41ed93867e4f77887"}
Sep 17 23:59:46.533516 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.533498    2580 scope.go:117] "RemoveContainer" containerID="ced41f7a0599a760bf0eccbdd348756ae96571f3a40e8074134702ec854301c3"
Sep 17 23:59:46.540798 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.540771    2580 scope.go:117] "RemoveContainer" containerID="b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5"
Sep 17 23:59:46.541311 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:59:46.541283    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5\": container with ID starting with b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5 not found: ID does not exist" containerID="b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5"
Sep 17 23:59:46.541389 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.541319    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5"} err="failed to get container status \"b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5\": rpc error: code = NotFound desc = could not find container \"b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5\": container with ID starting with b1f93a143aa47ae48d9b849a86d7887337c49a075453a85362f74f92b98906d5 not found: ID does not exist"
Sep 17 23:59:46.541389 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.541338    2580 scope.go:117] "RemoveContainer" containerID="ced41f7a0599a760bf0eccbdd348756ae96571f3a40e8074134702ec854301c3"
Sep 17 23:59:46.541577 ip-10-0-128-184 kubenswrapper[2580]: E0917 23:59:46.541562    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced41f7a0599a760bf0eccbdd348756ae96571f3a40e8074134702ec854301c3\": container with ID starting with ced41f7a0599a760bf0eccbdd348756ae96571f3a40e8074134702ec854301c3 not found: ID does not exist" containerID="ced41f7a0599a760bf0eccbdd348756ae96571f3a40e8074134702ec854301c3"
Sep 17 23:59:46.541691 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.541582    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced41f7a0599a760bf0eccbdd348756ae96571f3a40e8074134702ec854301c3"} err="failed to get container status \"ced41f7a0599a760bf0eccbdd348756ae96571f3a40e8074134702ec854301c3\": rpc error: code = NotFound desc = could not find container \"ced41f7a0599a760bf0eccbdd348756ae96571f3a40e8074134702ec854301c3\": container with ID starting with ced41f7a0599a760bf0eccbdd348756ae96571f3a40e8074134702ec854301c3 not found: ID does not exist"
Sep 17 23:59:46.566729 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.566707    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"]
Sep 17 23:59:46.570559 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:46.570513    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-dfb58c8f9-cjrlw"]
Sep 17 23:59:47.501668 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:47.501634    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0320fd0a-b61a-42a5-97a6-52296a7cb940" path="/var/lib/kubelet/pods/0320fd0a-b61a-42a5-97a6-52296a7cb940/volumes"
Sep 17 23:59:50.542060 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:50.542028    2580 generic.go:356] "Generic (PLEG): container finished" podID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerID="73fb6d9c63eb4737ea7d3cb1143a67eca41f403f325640660f648d528c4af825" exitCode=0
Sep 17 23:59:50.542060 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:50.542065    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" event={"ID":"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd","Type":"ContainerDied","Data":"73fb6d9c63eb4737ea7d3cb1143a67eca41f403f325640660f648d528c4af825"}
Sep 17 23:59:58.577896 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:58.577852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" event={"ID":"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd","Type":"ContainerStarted","Data":"41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704"}
Sep 17 23:59:58.578378 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:58.577969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" event={"ID":"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd","Type":"ContainerStarted","Data":"f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70"}
Sep 17 23:59:58.578378 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:58.578192    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:58.597313 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:58.597268    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podStartSLOduration=2.677393855 podStartE2EDuration="13.597254204s" totalImagesPullingTime="7.15918731s" totalInitContainerRuntime="3.760673039s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:59:58.595208861 +0000 UTC m=+2931.639585044" watchObservedRunningTime="2026-09-17 23:59:58.597254204 +0000 UTC m=+2931.641630364"
Sep 17 23:59:59.581361 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:59.581329    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 17 23:59:59.582352 ip-10-0-128-184 kubenswrapper[2580]: I0917 23:59:59.582328    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 00:00:00.585001 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:00:00.584963    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 00:00:05.590279 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:00:05.590247    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 18 00:00:05.590844 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:00:05.590818    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 00:00:15.591074 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:00:15.591039    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 00:00:25.591021 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:00:25.590978    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 00:00:35.590890 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:00:35.590794    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 00:00:45.591755 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:00:45.591716    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 00:00:55.591351 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:00:55.591309    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 00:01:05.590946 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:05.590907    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 00:01:15.591720 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:15.591683    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 00:01:23.500914 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:23.500883    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 18 00:01:26.879708 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:26.879676    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"]
Sep 18 00:01:26.880247 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:26.879913    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container" containerID="cri-o://f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70" gracePeriod=30
Sep 18 00:01:26.880247 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:26.879945    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kube-rbac-proxy" containerID="cri-o://41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704" gracePeriod=30
Sep 18 00:01:27.888140 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:27.888106    2580 generic.go:356] "Generic (PLEG): container finished" podID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerID="41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704" exitCode=2
Sep 18 00:01:27.888517 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:27.888161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" event={"ID":"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd","Type":"ContainerDied","Data":"41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704"}
Sep 18 00:01:30.585528 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:30.585486    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 18 00:01:31.221548 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.221527    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 18 00:01:31.312145 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.312120    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kube-rbac-proxy-sar-config\") pod \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") "
Sep 18 00:01:31.312287 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.312151    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxj45\" (UniqueName: \"kubernetes.io/projected/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kube-api-access-zxj45\") pod \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") "
Sep 18 00:01:31.312287 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.312186    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-proxy-tls\") pod \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") "
Sep 18 00:01:31.312287 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.312207    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kserve-provision-location\") pod \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") "
Sep 18 00:01:31.312287 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.312247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-cabundle-cert\") pod \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\" (UID: \"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd\") "
Sep 18 00:01:31.312556 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.312533    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" (UID: "3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:01:31.312640 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.312561    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" (UID: "3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:01:31.312640 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.312600    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" (UID: "3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:01:31.314278 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.314259    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kube-api-access-zxj45" (OuterVolumeSpecName: "kube-api-access-zxj45") pod "3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" (UID: "3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd"). InnerVolumeSpecName "kube-api-access-zxj45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:01:31.314376 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.314280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" (UID: "3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:01:31.413104 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.413009    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 18 00:01:31.413104 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.413071    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxj45\" (UniqueName: \"kubernetes.io/projected/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kube-api-access-zxj45\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 18 00:01:31.413104 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.413082    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 18 00:01:31.413104 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.413091    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 18 00:01:31.413104 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.413100    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 18 00:01:31.904833 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.904804    2580 generic.go:356] "Generic (PLEG): container finished" podID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerID="f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70" exitCode=0
Sep 18 00:01:31.905227 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.904852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" event={"ID":"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd","Type":"ContainerDied","Data":"f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70"}
Sep 18 00:01:31.905227 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.904860    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"
Sep 18 00:01:31.905227 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.904924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm" event={"ID":"3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd","Type":"ContainerDied","Data":"1998b72e667a29499bfc2288998256d5e861fc07cd7804889ea58c6c19846f8f"}
Sep 18 00:01:31.905227 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.904950    2580 scope.go:117] "RemoveContainer" containerID="41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704"
Sep 18 00:01:31.912658 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.912639    2580 scope.go:117] "RemoveContainer" containerID="f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70"
Sep 18 00:01:31.919868 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.919846    2580 scope.go:117] "RemoveContainer" containerID="73fb6d9c63eb4737ea7d3cb1143a67eca41f403f325640660f648d528c4af825"
Sep 18 00:01:31.924399 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.924377    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"]
Sep 18 00:01:31.927456 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.927435    2580 scope.go:117] "RemoveContainer" containerID="41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704"
Sep 18 00:01:31.927806 ip-10-0-128-184 kubenswrapper[2580]: E0918 00:01:31.927778    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704\": container with ID starting with 41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704 not found: ID does not exist" containerID="41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704"
Sep 18 00:01:31.927912 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.927811    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704"} err="failed to get container status \"41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704\": rpc error: code = NotFound desc = could not find container \"41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704\": container with ID starting with 41716689c19ccd2a487bbc35b708e99da1271a45a6be9dc77e9b00f3fb7e8704 not found: ID does not exist"
Sep 18 00:01:31.927912 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.927830    2580 scope.go:117] "RemoveContainer" containerID="f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70"
Sep 18 00:01:31.928024 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.927930    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p8khm"]
Sep 18 00:01:31.928145 ip-10-0-128-184 kubenswrapper[2580]: E0918 00:01:31.928128    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70\": container with ID starting with f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70 not found: ID does not exist" containerID="f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70"
Sep 18 00:01:31.928189 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.928153    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70"} err="failed to get container status \"f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70\": rpc error: code = NotFound desc = could not find container \"f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70\": container with ID starting with f7aff9d4d3fde7e92d017ae18ba36797942bc0400055671248883182d4cb6b70 not found: ID does not exist"
Sep 18 00:01:31.928189 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.928170    2580 scope.go:117] "RemoveContainer" containerID="73fb6d9c63eb4737ea7d3cb1143a67eca41f403f325640660f648d528c4af825"
Sep 18 00:01:31.928430 ip-10-0-128-184 kubenswrapper[2580]: E0918 00:01:31.928414    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73fb6d9c63eb4737ea7d3cb1143a67eca41f403f325640660f648d528c4af825\": container with ID starting with 73fb6d9c63eb4737ea7d3cb1143a67eca41f403f325640660f648d528c4af825 not found: ID does not exist" containerID="73fb6d9c63eb4737ea7d3cb1143a67eca41f403f325640660f648d528c4af825"
Sep 18 00:01:31.928479 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:31.928439    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fb6d9c63eb4737ea7d3cb1143a67eca41f403f325640660f648d528c4af825"} err="failed to get container status \"73fb6d9c63eb4737ea7d3cb1143a67eca41f403f325640660f648d528c4af825\": rpc error: code = NotFound desc = could not find container \"73fb6d9c63eb4737ea7d3cb1143a67eca41f403f325640660f648d528c4af825\": container with ID starting with 73fb6d9c63eb4737ea7d3cb1143a67eca41f403f325640660f648d528c4af825 not found: ID does not exist"
Sep 18 00:01:33.500308 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:01:33.500264    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" path="/var/lib/kubelet/pods/3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd/volumes"
Sep 18 00:03:56.820541 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820458    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"]
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820774    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0320fd0a-b61a-42a5-97a6-52296a7cb940" containerName="storage-initializer"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820788    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0320fd0a-b61a-42a5-97a6-52296a7cb940" containerName="storage-initializer"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820796    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0320fd0a-b61a-42a5-97a6-52296a7cb940" containerName="storage-initializer"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820801    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0320fd0a-b61a-42a5-97a6-52296a7cb940" containerName="storage-initializer"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820812    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kube-rbac-proxy"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820818    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kube-rbac-proxy"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820824    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="storage-initializer"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820829    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="storage-initializer"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820840    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820845    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820887    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kube-rbac-proxy"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820895    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0320fd0a-b61a-42a5-97a6-52296a7cb940" containerName="storage-initializer"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820900    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0320fd0a-b61a-42a5-97a6-52296a7cb940" containerName="storage-initializer"
Sep 18 00:03:56.821058 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.820905    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fbfa1b6-b513-4ad1-9b11-924d6e71bfbd" containerName="kserve-container"
Sep 18 00:03:56.823827 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.823811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:56.825945 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.825920    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b53da9-predictor-serving-cert\""
Sep 18 00:03:56.826099 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.825989    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b53da9-kube-rbac-proxy-sar-config\""
Sep 18 00:03:56.826099 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.826001    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-b53da9-dockercfg-wqvlx\""
Sep 18 00:03:56.826099 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.826079    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-b53da9\""
Sep 18 00:03:56.826284 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.826083    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 00:03:56.826921 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.826900    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:03:56.827035 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.826935    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 00:03:56.834940 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.834917    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"]
Sep 18 00:03:56.963045 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.963005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba49ac6-5c22-468d-952e-80c311a0a52e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:56.963045 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.963045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba49ac6-5c22-468d-952e-80c311a0a52e-kserve-provision-location\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:56.963292 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.963067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4ppg\" (UniqueName: \"kubernetes.io/projected/dba49ac6-5c22-468d-952e-80c311a0a52e-kube-api-access-x4ppg\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:56.963292 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.963146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba49ac6-5c22-468d-952e-80c311a0a52e-proxy-tls\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:56.963292 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:56.963192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba49ac6-5c22-468d-952e-80c311a0a52e-cabundle-cert\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:57.064236 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:57.064199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba49ac6-5c22-468d-952e-80c311a0a52e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:57.064236 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:57.064239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba49ac6-5c22-468d-952e-80c311a0a52e-kserve-provision-location\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:57.064482 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:57.064258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4ppg\" (UniqueName: \"kubernetes.io/projected/dba49ac6-5c22-468d-952e-80c311a0a52e-kube-api-access-x4ppg\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:57.064482 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:57.064277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba49ac6-5c22-468d-952e-80c311a0a52e-proxy-tls\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:57.064482 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:57.064321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba49ac6-5c22-468d-952e-80c311a0a52e-cabundle-cert\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:57.064720 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:57.064699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba49ac6-5c22-468d-952e-80c311a0a52e-kserve-provision-location\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:57.065035 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:57.065012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba49ac6-5c22-468d-952e-80c311a0a52e-cabundle-cert\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:57.065106 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:57.065014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba49ac6-5c22-468d-952e-80c311a0a52e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:57.066938 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:57.066922    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba49ac6-5c22-468d-952e-80c311a0a52e-proxy-tls\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:57.079482 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:57.079405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4ppg\" (UniqueName: \"kubernetes.io/projected/dba49ac6-5c22-468d-952e-80c311a0a52e-kube-api-access-x4ppg\") pod \"isvc-secondary-b53da9-predictor-5664c4694f-lclzk\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") " pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:57.133245 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:57.133203    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:03:57.407924 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:57.407831    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"]
Sep 18 00:03:57.411517 ip-10-0-128-184 kubenswrapper[2580]: W0918 00:03:57.411488    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba49ac6_5c22_468d_952e_80c311a0a52e.slice/crio-db408366a8f833e719e376b9cb194db0524b581cdce80fc49f42c0b755eb9d13 WatchSource:0}: Error finding container db408366a8f833e719e376b9cb194db0524b581cdce80fc49f42c0b755eb9d13: Status 404 returned error can't find the container with id db408366a8f833e719e376b9cb194db0524b581cdce80fc49f42c0b755eb9d13
Sep 18 00:03:58.397038 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:58.396996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk" event={"ID":"dba49ac6-5c22-468d-952e-80c311a0a52e","Type":"ContainerStarted","Data":"dc41617dc925f2d876d45deced6e43ebc4f81a2278a2a679006fca68da5b57ae"}
Sep 18 00:03:58.397427 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:03:58.397090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk" event={"ID":"dba49ac6-5c22-468d-952e-80c311a0a52e","Type":"ContainerStarted","Data":"db408366a8f833e719e376b9cb194db0524b581cdce80fc49f42c0b755eb9d13"}
Sep 18 00:04:04.418810 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:04.418783    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b53da9-predictor-5664c4694f-lclzk_dba49ac6-5c22-468d-952e-80c311a0a52e/storage-initializer/0.log"
Sep 18 00:04:04.419187 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:04.418820    2580 generic.go:356] "Generic (PLEG): container finished" podID="dba49ac6-5c22-468d-952e-80c311a0a52e" containerID="dc41617dc925f2d876d45deced6e43ebc4f81a2278a2a679006fca68da5b57ae" exitCode=1
Sep 18 00:04:04.419187 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:04.418857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk" event={"ID":"dba49ac6-5c22-468d-952e-80c311a0a52e","Type":"ContainerDied","Data":"dc41617dc925f2d876d45deced6e43ebc4f81a2278a2a679006fca68da5b57ae"}
Sep 18 00:04:05.424150 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:05.424124    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b53da9-predictor-5664c4694f-lclzk_dba49ac6-5c22-468d-952e-80c311a0a52e/storage-initializer/0.log"
Sep 18 00:04:05.424552 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:05.424199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk" event={"ID":"dba49ac6-5c22-468d-952e-80c311a0a52e","Type":"ContainerStarted","Data":"b5ab1fad3dd2ed8d56573d1cb5f980a4f21fdd3ca2f3d47f9fbac74e51a5dae0"}
Sep 18 00:04:08.437088 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:08.437056    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b53da9-predictor-5664c4694f-lclzk_dba49ac6-5c22-468d-952e-80c311a0a52e/storage-initializer/1.log"
Sep 18 00:04:08.437480 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:08.437465    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b53da9-predictor-5664c4694f-lclzk_dba49ac6-5c22-468d-952e-80c311a0a52e/storage-initializer/0.log"
Sep 18 00:04:08.437530 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:08.437497    2580 generic.go:356] "Generic (PLEG): container finished" podID="dba49ac6-5c22-468d-952e-80c311a0a52e" containerID="b5ab1fad3dd2ed8d56573d1cb5f980a4f21fdd3ca2f3d47f9fbac74e51a5dae0" exitCode=1
Sep 18 00:04:08.437584 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:08.437526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk" event={"ID":"dba49ac6-5c22-468d-952e-80c311a0a52e","Type":"ContainerDied","Data":"b5ab1fad3dd2ed8d56573d1cb5f980a4f21fdd3ca2f3d47f9fbac74e51a5dae0"}
Sep 18 00:04:08.437584 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:08.437581    2580 scope.go:117] "RemoveContainer" containerID="dc41617dc925f2d876d45deced6e43ebc4f81a2278a2a679006fca68da5b57ae"
Sep 18 00:04:08.437961 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:08.437943    2580 scope.go:117] "RemoveContainer" containerID="dc41617dc925f2d876d45deced6e43ebc4f81a2278a2a679006fca68da5b57ae"
Sep 18 00:04:08.448488 ip-10-0-128-184 kubenswrapper[2580]: E0918 00:04:08.448461    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-b53da9-predictor-5664c4694f-lclzk_kserve-ci-e2e-test(dba49ac6-5c22-468d-952e-80c311a0a52e)\"" pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk" podUID="dba49ac6-5c22-468d-952e-80c311a0a52e"
Sep 18 00:04:08.448748 ip-10-0-128-184 kubenswrapper[2580]: E0918 00:04:08.448720    2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b53da9-predictor-5664c4694f-lclzk_kserve-ci-e2e-test_dba49ac6-5c22-468d-952e-80c311a0a52e_0 in pod sandbox db408366a8f833e719e376b9cb194db0524b581cdce80fc49f42c0b755eb9d13 from index: no such id: 'dc41617dc925f2d876d45deced6e43ebc4f81a2278a2a679006fca68da5b57ae'" containerID="dc41617dc925f2d876d45deced6e43ebc4f81a2278a2a679006fca68da5b57ae"
Sep 18 00:04:08.448825 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:08.448757    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc41617dc925f2d876d45deced6e43ebc4f81a2278a2a679006fca68da5b57ae"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b53da9-predictor-5664c4694f-lclzk_kserve-ci-e2e-test_dba49ac6-5c22-468d-952e-80c311a0a52e_0 in pod sandbox db408366a8f833e719e376b9cb194db0524b581cdce80fc49f42c0b755eb9d13 from index: no such id: 'dc41617dc925f2d876d45deced6e43ebc4f81a2278a2a679006fca68da5b57ae'"
Sep 18 00:04:09.442778 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:09.442748    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b53da9-predictor-5664c4694f-lclzk_dba49ac6-5c22-468d-952e-80c311a0a52e/storage-initializer/1.log"
Sep 18 00:04:14.831007 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.830980    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"]
Sep 18 00:04:14.962060 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.962040    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b53da9-predictor-5664c4694f-lclzk_dba49ac6-5c22-468d-952e-80c311a0a52e/storage-initializer/1.log"
Sep 18 00:04:14.965802 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.965786    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:04:14.997418 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.997380    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba49ac6-5c22-468d-952e-80c311a0a52e-cabundle-cert\") pod \"dba49ac6-5c22-468d-952e-80c311a0a52e\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") "
Sep 18 00:04:14.997418 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.997425    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba49ac6-5c22-468d-952e-80c311a0a52e-kube-rbac-proxy-sar-config\") pod \"dba49ac6-5c22-468d-952e-80c311a0a52e\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") "
Sep 18 00:04:14.997653 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.997484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4ppg\" (UniqueName: \"kubernetes.io/projected/dba49ac6-5c22-468d-952e-80c311a0a52e-kube-api-access-x4ppg\") pod \"dba49ac6-5c22-468d-952e-80c311a0a52e\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") "
Sep 18 00:04:14.997653 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.997504    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba49ac6-5c22-468d-952e-80c311a0a52e-kserve-provision-location\") pod \"dba49ac6-5c22-468d-952e-80c311a0a52e\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") "
Sep 18 00:04:14.997653 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.997554    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba49ac6-5c22-468d-952e-80c311a0a52e-proxy-tls\") pod \"dba49ac6-5c22-468d-952e-80c311a0a52e\" (UID: \"dba49ac6-5c22-468d-952e-80c311a0a52e\") "
Sep 18 00:04:14.997881 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.997849    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba49ac6-5c22-468d-952e-80c311a0a52e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dba49ac6-5c22-468d-952e-80c311a0a52e" (UID: "dba49ac6-5c22-468d-952e-80c311a0a52e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:04:14.997955 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.997874    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba49ac6-5c22-468d-952e-80c311a0a52e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dba49ac6-5c22-468d-952e-80c311a0a52e" (UID: "dba49ac6-5c22-468d-952e-80c311a0a52e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:04:14.997955 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.997878    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba49ac6-5c22-468d-952e-80c311a0a52e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dba49ac6-5c22-468d-952e-80c311a0a52e" (UID: "dba49ac6-5c22-468d-952e-80c311a0a52e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:04:14.999815 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.999790    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba49ac6-5c22-468d-952e-80c311a0a52e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dba49ac6-5c22-468d-952e-80c311a0a52e" (UID: "dba49ac6-5c22-468d-952e-80c311a0a52e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:04:14.999907 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:14.999881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba49ac6-5c22-468d-952e-80c311a0a52e-kube-api-access-x4ppg" (OuterVolumeSpecName: "kube-api-access-x4ppg") pod "dba49ac6-5c22-468d-952e-80c311a0a52e" (UID: "dba49ac6-5c22-468d-952e-80c311a0a52e"). InnerVolumeSpecName "kube-api-access-x4ppg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:04:15.098473 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:15.098375    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4ppg\" (UniqueName: \"kubernetes.io/projected/dba49ac6-5c22-468d-952e-80c311a0a52e-kube-api-access-x4ppg\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 18 00:04:15.098473 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:15.098407    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba49ac6-5c22-468d-952e-80c311a0a52e-kserve-provision-location\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 18 00:04:15.098473 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:15.098421    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba49ac6-5c22-468d-952e-80c311a0a52e-proxy-tls\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 18 00:04:15.098473 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:15.098433    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba49ac6-5c22-468d-952e-80c311a0a52e-cabundle-cert\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 18 00:04:15.098473 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:15.098444    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba49ac6-5c22-468d-952e-80c311a0a52e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-184.ec2.internal\" DevicePath \"\""
Sep 18 00:04:15.465127 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:15.465047    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b53da9-predictor-5664c4694f-lclzk_dba49ac6-5c22-468d-952e-80c311a0a52e/storage-initializer/1.log"
Sep 18 00:04:15.465305 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:15.465145    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"
Sep 18 00:04:15.465305 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:15.465167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk" event={"ID":"dba49ac6-5c22-468d-952e-80c311a0a52e","Type":"ContainerDied","Data":"db408366a8f833e719e376b9cb194db0524b581cdce80fc49f42c0b755eb9d13"}
Sep 18 00:04:15.465305 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:15.465269    2580 scope.go:117] "RemoveContainer" containerID="b5ab1fad3dd2ed8d56573d1cb5f980a4f21fdd3ca2f3d47f9fbac74e51a5dae0"
Sep 18 00:04:15.505202 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:15.505169    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"]
Sep 18 00:04:15.509345 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:15.509319    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b53da9-predictor-5664c4694f-lclzk"]
Sep 18 00:04:17.500350 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:04:17.500318    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dba49ac6-5c22-468d-952e-80c311a0a52e" path="/var/lib/kubelet/pods/dba49ac6-5c22-468d-952e-80c311a0a52e/volumes"
Sep 18 00:05:00.935955 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:00.935879    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xt5fb_0bc41eaf-0a42-4187-8eec-c27fd3ab26d8/global-pull-secret-syncer/0.log"
Sep 18 00:05:01.087389 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:01.087357    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pgkdk_70431036-0857-4c0f-83f5-842c5b84a114/konnectivity-agent/0.log"
Sep 18 00:05:01.166076 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:01.166041    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-184.ec2.internal_3b0094ac69bc5cb13f6f1d4e50931ad3/haproxy/0.log"
Sep 18 00:05:05.552278 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:05.552250    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-csnjv_1913439d-3dd6-448f-8e13-31fecc7456a2/node-exporter/0.log"
Sep 18 00:05:05.580223 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:05.580194    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-csnjv_1913439d-3dd6-448f-8e13-31fecc7456a2/kube-rbac-proxy/0.log"
Sep 18 00:05:05.611333 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:05.611306    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-csnjv_1913439d-3dd6-448f-8e13-31fecc7456a2/init-textfile/0.log"
Sep 18 00:05:09.036014 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.035976    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"]
Sep 18 00:05:09.036442 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.036242    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba49ac6-5c22-468d-952e-80c311a0a52e" containerName="storage-initializer"
Sep 18 00:05:09.036442 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.036253    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba49ac6-5c22-468d-952e-80c311a0a52e" containerName="storage-initializer"
Sep 18 00:05:09.036442 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.036307    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba49ac6-5c22-468d-952e-80c311a0a52e" containerName="storage-initializer"
Sep 18 00:05:09.040188 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.040173    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.042473 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.042442    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skn79\"/\"openshift-service-ca.crt\""
Sep 18 00:05:09.042650 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.042519    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skn79\"/\"kube-root-ca.crt\""
Sep 18 00:05:09.043413 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.043394    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-skn79\"/\"default-dockercfg-q8r7t\""
Sep 18 00:05:09.049182 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.049160    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"]
Sep 18 00:05:09.195453 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.195421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-sys\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.195453 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.195458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-lib-modules\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.195759 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.195486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-proc\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.195759 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.195503    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-podres\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.195759 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.195518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2mls\" (UniqueName: \"kubernetes.io/projected/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-kube-api-access-j2mls\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.296450 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.296372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-sys\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.296450 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.296408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-lib-modules\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.296450 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.296436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-proc\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.296662 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.296460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-podres\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.296662 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.296501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-sys\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.296662 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.296517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-proc\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.296662 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.296550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2mls\" (UniqueName: \"kubernetes.io/projected/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-kube-api-access-j2mls\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.296662 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.296563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-podres\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.296662 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.296564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-lib-modules\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.305396 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.305379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2mls\" (UniqueName: \"kubernetes.io/projected/bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f-kube-api-access-j2mls\") pod \"perf-node-gather-daemonset-xghbl\" (UID: \"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.349576 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.349541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:09.620161 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.620081    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"]
Sep 18 00:05:09.623086 ip-10-0-128-184 kubenswrapper[2580]: W0918 00:05:09.623056    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbc4b4c0e_bab7_4fe0_8825_ad85cbd0578f.slice/crio-8d5ce06e58ff2b9208cb277192f537d89186a9606fc9c2dfa30f5d20c50ee166 WatchSource:0}: Error finding container 8d5ce06e58ff2b9208cb277192f537d89186a9606fc9c2dfa30f5d20c50ee166: Status 404 returned error can't find the container with id 8d5ce06e58ff2b9208cb277192f537d89186a9606fc9c2dfa30f5d20c50ee166
Sep 18 00:05:09.658154 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:09.658128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl" event={"ID":"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f","Type":"ContainerStarted","Data":"8d5ce06e58ff2b9208cb277192f537d89186a9606fc9c2dfa30f5d20c50ee166"}
Sep 18 00:05:10.393203 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:10.393172    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x8v6t_6be6dc27-bc71-4d36-a57f-7f3b8557d0c3/dns/0.log"
Sep 18 00:05:10.417247 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:10.417227    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x8v6t_6be6dc27-bc71-4d36-a57f-7f3b8557d0c3/kube-rbac-proxy/0.log"
Sep 18 00:05:10.468690 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:10.468658    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t65zf_60bb6bc0-7d80-4b8a-8d28-b27636b5e7a5/dns-node-resolver/0.log"
Sep 18 00:05:10.663537 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:10.663451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl" event={"ID":"bc4b4c0e-bab7-4fe0-8825-ad85cbd0578f","Type":"ContainerStarted","Data":"030e4a93d1d08583dd533c3a7ef4befca565a078f19543a96d4a501d44ef4d7c"}
Sep 18 00:05:10.663706 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:10.663577    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:10.681493 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:10.681448    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl" podStartSLOduration=1.681434369 podStartE2EDuration="1.681434369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 00:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:05:10.679763884 +0000 UTC m=+3243.724140044" watchObservedRunningTime="2026-09-18 00:05:10.681434369 +0000 UTC m=+3243.725810529"
Sep 18 00:05:11.036379 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:11.036346    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kcvpd_265e74c8-a31a-43de-af18-711758cc8e5d/node-ca/0.log"
Sep 18 00:05:12.184695 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:12.184672    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rj6f4_11ed4035-5836-4a4a-ab0e-d73776ada4cf/serve-healthcheck-canary/0.log"
Sep 18 00:05:12.658122 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:12.658091    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-72k7x_924de91b-c17e-4d3b-933c-f8129c8b3265/kube-rbac-proxy/0.log"
Sep 18 00:05:12.680097 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:12.680063    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-72k7x_924de91b-c17e-4d3b-933c-f8129c8b3265/exporter/0.log"
Sep 18 00:05:12.705069 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:12.705035    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-72k7x_924de91b-c17e-4d3b-933c-f8129c8b3265/extractor/0.log"
Sep 18 00:05:15.113029 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:15.112999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-rqr2q_50ae5543-eb37-4b06-8384-f7db8e0df56a/manager/0.log"
Sep 18 00:05:15.623235 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:15.623202    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-8smpb_0137c215-8ab6-4514-bf18-b8bd1934d8d3/manager/0.log"
Sep 18 00:05:16.677248 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:16.677216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-xghbl"
Sep 18 00:05:21.463452 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:21.463422    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2p4f8_405ad021-9316-4805-b71b-3a6a13854a1d/kube-multus-additional-cni-plugins/0.log"
Sep 18 00:05:21.498041 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:21.498011    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2p4f8_405ad021-9316-4805-b71b-3a6a13854a1d/egress-router-binary-copy/0.log"
Sep 18 00:05:21.520249 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:21.520221    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2p4f8_405ad021-9316-4805-b71b-3a6a13854a1d/cni-plugins/0.log"
Sep 18 00:05:21.546051 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:21.546029    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2p4f8_405ad021-9316-4805-b71b-3a6a13854a1d/bond-cni-plugin/0.log"
Sep 18 00:05:21.570121 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:21.570081    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2p4f8_405ad021-9316-4805-b71b-3a6a13854a1d/routeoverride-cni/0.log"
Sep 18 00:05:21.593218 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:21.593188    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2p4f8_405ad021-9316-4805-b71b-3a6a13854a1d/whereabouts-cni-bincopy/0.log"
Sep 18 00:05:21.616731 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:21.616710    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2p4f8_405ad021-9316-4805-b71b-3a6a13854a1d/whereabouts-cni/0.log"
Sep 18 00:05:22.093017 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:22.092993    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zgnlk_ff87e86b-3f0f-4ab5-ba79-86dae21e2c95/kube-multus/0.log"
Sep 18 00:05:22.118357 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:22.118332    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gr7p6_1e6da33b-9d1e-472a-97aa-e4951d3972ff/network-metrics-daemon/0.log"
Sep 18 00:05:22.140601 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:22.140574    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gr7p6_1e6da33b-9d1e-472a-97aa-e4951d3972ff/kube-rbac-proxy/0.log"
Sep 18 00:05:23.266838 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:23.266811    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jnsbp_bbd99a55-d6c1-44c0-adda-255aab5fb532/ovn-controller/0.log"
Sep 18 00:05:23.285367 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:23.285341    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jnsbp_bbd99a55-d6c1-44c0-adda-255aab5fb532/ovn-acl-logging/0.log"
Sep 18 00:05:23.312855 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:23.312833    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jnsbp_bbd99a55-d6c1-44c0-adda-255aab5fb532/kube-rbac-proxy-node/0.log"
Sep 18 00:05:23.335598 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:23.335570    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jnsbp_bbd99a55-d6c1-44c0-adda-255aab5fb532/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 00:05:23.355150 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:23.355126    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jnsbp_bbd99a55-d6c1-44c0-adda-255aab5fb532/northd/0.log"
Sep 18 00:05:23.376409 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:23.376379    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jnsbp_bbd99a55-d6c1-44c0-adda-255aab5fb532/nbdb/0.log"
Sep 18 00:05:23.397674 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:23.397650    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jnsbp_bbd99a55-d6c1-44c0-adda-255aab5fb532/sbdb/0.log"
Sep 18 00:05:23.458407 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:23.458381    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jnsbp_bbd99a55-d6c1-44c0-adda-255aab5fb532/ovnkube-controller/0.log"
Sep 18 00:05:24.710966 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:24.710883    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-s6b9s_cac347e3-b53f-4871-a2c2-4c536a2fa04c/check-endpoints/0.log"
Sep 18 00:05:24.740069 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:24.740045    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-8gs2j_50270559-ac36-4cd1-85e4-1267baf1b3cc/network-check-target-container/0.log"
Sep 18 00:05:25.788520 ip-10-0-128-184 kubenswrapper[2580]: I0918 00:05:25.788491    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zn8l4_39469e70-5457-494d-9f87-7cf3f8c7d557/iptables-alerter/0.log"