Sep 16 18:45:16.062355 ip-10-0-135-44 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:45:16.495416 ip-10-0-135-44 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:16.495416 ip-10-0-135-44 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:45:16.495416 ip-10-0-135-44 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:16.495416 ip-10-0-135-44 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:45:16.495416 ip-10-0-135-44 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:16.498186 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.498128    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:45:16.501211 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501197    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:16.501211 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501210    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501214    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501217    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501220    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501223    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501226    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501228    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501231    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501237    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501240    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501242    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501245    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501247    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501250    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501252    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501255    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501257    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501262    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501265    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:16.501273 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501268    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501271    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501274    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501277    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501280    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501283    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501286    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501288    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501291    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501294    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501297    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501301    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501303    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501306    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501308    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501311    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501313    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501316    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501319    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:16.501765 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501321    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501324    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501326    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501329    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501331    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501334    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501336    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501338    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501341    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501343    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501345    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501349    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501351    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501354    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501356    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501359    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501362    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501364    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501367    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501369    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:16.502466 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501372    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501376    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501379    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501382    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501385    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501388    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501390    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501394    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501396    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501399    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501401    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501404    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501408    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501410    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501413    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501415    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501417    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501420    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501422    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:16.502954 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501425    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501427    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501429    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501432    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501434    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501436    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501439    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501441    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501446    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501448    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501451    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501453    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501455    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501458    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501461    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501463    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501465    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501468    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501470    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501473    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:16.503431 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.501475    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:16.504651 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504635    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:16.504651 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504651    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504655    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504658    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504661    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504664    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504667    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504670    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504673    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504676    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504679    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504682    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504684    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504687    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504691    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504695    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504698    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504700    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504703    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504705    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504708    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:16.504711 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504711    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504714    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504716    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504719    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504722    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504725    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504727    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504730    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504733    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504737    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504740    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504743    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504746    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504748    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504752    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504754    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504757    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504760    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504763    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:16.505205 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504765    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504768    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504770    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504773    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504775    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504778    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504780    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504782    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504785    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504787    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504790    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504792    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504795    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504798    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504801    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504804    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504807    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504809    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504812    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504816    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:16.505686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504820    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504822    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504825    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504827    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504830    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504832    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504834    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504837    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504839    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504842    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504844    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504847    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504849    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504852    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504854    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504857    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504859    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504862    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504865    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504868    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:16.506176 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504870    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504873    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504875    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504878    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504880    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504883    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504887    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504890    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504893    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504895    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504898    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504901    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504903    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504906    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504908    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504910    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504913    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504916    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.504918    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505262    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:16.506650 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505267    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505270    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505273    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505275    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505278    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505281    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505284    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505286    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505288    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505291    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505293    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505296    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505299    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505301    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505304    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505307    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505310    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505312    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505315    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505317    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:16.507155 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505320    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505323    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505325    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505328    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505330    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505332    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505335    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505338    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505340    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505342    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505345    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505347    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505350    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505352    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505354    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505357    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505359    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505363    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505366    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505369    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:16.507648 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505371    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505374    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505376    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505378    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505381    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505383    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505386    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505388    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505391    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505394    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505397    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505399    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505402    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505404    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505407    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505409    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505411    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505414    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505416    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505419    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:16.508132 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505421    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505423    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505426    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505428    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505432    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505435    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505438    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505441    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505444    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505447    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505449    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505452    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505454    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505458    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505460    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505463    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505466    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505468    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505471    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:16.508617 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505474    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505476    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505480    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505483    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505485    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505487    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505490    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505493    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505495    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505497    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505500    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505502    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505505    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505507    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505511    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505514    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505517    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505520    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.505522    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505599    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:45:16.509077 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505642    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505649    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505654    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505657    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505661    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505665    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505669    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505672    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505675    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505678    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505681    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505685    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505688    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505691    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505694    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505697    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505701    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505704    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505707    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505710    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505713    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505718    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505721    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:45:16.509567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505724    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505727    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505730    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505733    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505736    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505739    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505742    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505744    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505748    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505751    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505754    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505756    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505759    2565 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505762    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505766    2565 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505769    2565 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505772    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505775    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505778    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505781    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505784    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505787    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505790    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505793    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505796    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:45:16.510126 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505799    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505802    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505804    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505807    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505809    2565 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505815    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505818    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505821    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505824    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505827    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505830    2565 flags.go:64] FLAG: --help="false"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505832    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505835    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505838    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505841    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505844    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505847    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505850    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505853    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505855    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505861    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505864    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505867    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505869    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:45:16.510765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505872    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505875    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505878    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505881    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505883    2565 flags.go:64] FLAG: --lock-file=""
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505886    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505889    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505891    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505897    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505900    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505903    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505905    2565 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505908    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505912    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505915    2565 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505918    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505922    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505925    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505928    2565 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505931    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505934    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505937    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505939    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505942    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505945    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:45:16.511371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505947    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505954    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505957    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505960    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505963    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505966    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505971    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505974    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505977    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505979    2565 flags.go:64] FLAG: --port="10250"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505982    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505985    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f4108032fda199bb"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505988    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505991    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505994    2565 flags.go:64] FLAG: --register-node="true"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.505997    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506000    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506003    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506006    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506009    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506013    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506016    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506019    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506022    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506024    2565 flags.go:64] FLAG: --runonce="false"
Sep 16 18:45:16.511970 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506027    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506031    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506034    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506037    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506039    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506042    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506045    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506048    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506051    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506054    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506056    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506060    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506063    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506066    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506069    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506074    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506076    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506079    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506082    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506085    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506088    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506091    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506094    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506097    2565 flags.go:64] FLAG: --v="2"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506100    2565 flags.go:64] FLAG: --version="false"
Sep 16 18:45:16.512599 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506116    2565 flags.go:64] FLAG: --vmodule=""
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506120    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.506124    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506227    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506231    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506234    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506237    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506239    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506242    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506245    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506247    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506250    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506252    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506255    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506257    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506260    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506262    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506264    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506268    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506271    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:16.513265 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506273    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506276    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506279    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506281    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506284    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506287    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506291    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506294    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506297    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506300    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506303    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506305    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506308    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506311    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506316    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506319    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506322    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506325    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506327    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506330    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:16.513746 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506332    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506335    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506339    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506342    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506345    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506347    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506350    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506352    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506355    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506357    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506361    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506363    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506366    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506368    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506371    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506374    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506376    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506379    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506381    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506384    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:16.514259 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506386    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506389    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506392    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506395    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506398    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506401    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506406    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506409    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506412    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506415    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506418    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506420    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506423    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506426    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506428    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506431    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506434    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506436    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506439    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:16.514770 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506441    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506444    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506446    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506450    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506452    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506455    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506458    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506460    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506463    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506465    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506468    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506470    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506472    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506475    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506477    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506480    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506482    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506485    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506487    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506491    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:16.515257 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506494    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506497    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.506499    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.507395    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.514303    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.514317    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.514333    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.514338    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514400    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514404    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514408    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514411    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514414    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514417    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514420    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:16.515737 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514422    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514425    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514428    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514432    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514436    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514439    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514442    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514445    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514447    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514450    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514454    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514457    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514459    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514462    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514465    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514467    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514470    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514473    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514476    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:16.516116 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514478    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514480    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514484    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514486    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514489    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514492    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514494    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514497    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514499    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514501    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514504    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514506    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514508    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514511    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514513    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514516    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514518    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514521    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514523    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514526    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:16.516562 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514528    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514531    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514533    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514536    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514538    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514540    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514543    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514546    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514548    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514550    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514553    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514557    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514560    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514563    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514566    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514569    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514571    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514574    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514576    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:16.517026 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514579    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514581    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514584    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514586    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514588    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514591    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514593    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514596    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514598    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514601    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514603    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514606    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514608    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514612    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514616    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514618    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514621    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514623    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514626    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514628    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:16.517504 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514631    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514633    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514636    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514638    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514640    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514643    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514646    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514648    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514651    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514654    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514656    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514658    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514661    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514663    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.514668    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:16.518020 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514782    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514787    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514790    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514793    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514796    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514799    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514802    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514804    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514807    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514810    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514812    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514815    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514817    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514820    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514822    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514825    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514828    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514830    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514832    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514835    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:16.518392 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514837    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514840    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514843    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514846    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514848    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514851    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514854    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514856    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514859    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514862    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514864    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514867    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514876    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514879    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514882    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514884    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514887    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514889    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514892    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:16.518864 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514894    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514897    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514899    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514903    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514906    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514910    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514913    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514916    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514919    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514921    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514924    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514927    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514929    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514932    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514934    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514937    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514939    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514943    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514946    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:16.519342 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514948    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514951    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514955    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514957    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514960    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514963    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514966    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514968    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514970    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514973    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514975    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514978    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514980    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514983    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514985    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514987    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514990    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514992    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514995    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.514997    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:16.519791 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515000    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515002    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515005    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515007    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515009    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515012    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515014    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515017    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515019    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515021    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515024    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515027    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515030    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515032    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515035    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515037    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515039    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515042    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515045    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515047    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:16.520281 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:16.515050    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:16.520755 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.515054    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:16.520755 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.515161    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:45:16.520755 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.517546    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:45:16.520755 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.518530    2565 server.go:1019] "Starting client certificate rotation"
Sep 16 18:45:16.520755 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.518659    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:16.520755 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.518719    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:16.521415 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.521402    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:16.522617 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.522599    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:16.535634 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.535616    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:45:16.538811 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.538794    2565 log.go:25] "Validated CRI v1 image API"
Sep 16 18:45:16.539568 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.539555    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:45:16.542821 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.542805    2565 fs.go:135] Filesystem UUIDs: map[0f1aa267-6ed6-4f38-9a0b-64ade0047468:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f20ea40b-2d13-46f1-aa28-3a221bffdf4f:/dev/nvme0n1p4]
Sep 16 18:45:16.542879 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.542822    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:45:16.548731 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.548639    2565 manager.go:216] Machine: {Timestamp:2026-09-16 18:45:16.547368553 +0000 UTC m=+0.360957402 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200146 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22db91cfbe5e89d03144d06984b4ab SystemUUID:ec22db91-cfbe-5e89-d031-44d06984b4ab BootID:47da29f7-4253-4461-b163-1aee437524e3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f5:ce:a2:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f5:ce:a2:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:9e:e8:0b:d5:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:45:16.548731 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.548726    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:45:16.548826 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.548787    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:45:16.551283 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.551267    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:16.552246 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.552223    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:45:16.552373 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.552250    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-44.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:45:16.553136 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.553126    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:45:16.553177 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.553138    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:45:16.553177 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.553146    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:16.554117 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.554097    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:16.555328 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.555318    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:45:16.559996 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.559985    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:16.560091 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.560082    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:45:16.560130 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.560122    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:45:16.560163 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.560132    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:45:16.560163 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.560145    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:45:16.560163 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.560155    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:45:16.560163 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.560163    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:45:16.560453 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.560438    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:16.560505 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.560459    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:16.562753 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.562739    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:45:16.565681 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.565666    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:45:16.565729 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.565713    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:45:16.567757 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.567746    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:45:16.567789 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.567763    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:45:16.567789 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.567769    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:45:16.567789 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.567774    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:45:16.567789 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.567782    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:45:16.567789 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.567787    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:45:16.567916 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.567801    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:45:16.567916 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.567806    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:45:16.567916 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.567812    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:45:16.567916 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.567819    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:45:16.567916 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.567832    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:45:16.567916 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.567838    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:45:16.568773 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.568763    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:45:16.568773 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.568773    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:45:16.573814 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.573797    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-44.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:45:16.573895 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.573884    2565 server.go:1268] "Started kubelet"
Sep 16 18:45:16.574012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.573960    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:45:16.574127 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.573978    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:45:16.574127 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.574054    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:45:16.574320 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.574250    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-44.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:45:16.574381 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.574346    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:45:16.574611 ip-10-0-135-44 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:45:16.575623 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.575534    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:45:16.576571 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.576556    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:45:16.578737 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.578718    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:45:16.582771 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.582758    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:45:16.582837 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.582771    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:16.582893 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.582869    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:16.583400 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.582218    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-44.ec2.internal.18d5e1744cfe92db  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-44.ec2.internal,UID:ip-10-0-135-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-44.ec2.internal,},FirstTimestamp:2026-09-16 18:45:16.573848283 +0000 UTC m=+0.387437133,LastTimestamp:2026-09-16 18:45:16.573848283 +0000 UTC m=+0.387437133,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-44.ec2.internal,}"
Sep 16 18:45:16.583500 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.583453    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:45:16.583500 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.583459    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:45:16.583500 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.583476    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:45:16.583626 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.583572    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:45:16.583626 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.583595    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:45:16.583626 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.583609    2565 factory.go:55] Registering systemd factory
Sep 16 18:45:16.583626 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.583617    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:45:16.583772 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.583600    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:45:16.583818 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.583802    2565 factory.go:153] Registering CRI-O factory
Sep 16 18:45:16.583818 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.583814    2565 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:45:16.583902 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.583828    2565 factory.go:103] Registering Raw factory
Sep 16 18:45:16.583902 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.583848    2565 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:45:16.584194 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.584181    2565 manager.go:349] Starting recovery of all containers
Sep 16 18:45:16.588448 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.588422    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-44.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:45:16.588528 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.588512    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:45:16.591122 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.591064    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:45:16.596631 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.596613    2565 manager.go:354] Recovery completed
Sep 16 18:45:16.600953 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.600939    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:16.602758 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.602736    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:16.602819 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.602770    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:16.602819 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.602780    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:16.603281 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.603266    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:45:16.603281 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.603280    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:45:16.603390 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.603298    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:16.603390 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.603281    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:16.604141 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.604121    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:16.604230 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.604148    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:16.604230 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.604165    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:16.605090 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.604997    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-44.ec2.internal.18d5e1744eb7b408  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-44.ec2.internal,UID:ip-10-0-135-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-44.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-44.ec2.internal,},FirstTimestamp:2026-09-16 18:45:16.602758152 +0000 UTC m=+0.416347001,LastTimestamp:2026-09-16 18:45:16.602758152 +0000 UTC m=+0.416347001,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-44.ec2.internal,}"
Sep 16 18:45:16.605444 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.605426    2565 policy_none.go:49] "None policy: Start"
Sep 16 18:45:16.605444 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.605445    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:45:16.605591 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.605454    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:45:16.607384 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.607371    2565 policy_none.go:47] "Start"
Sep 16 18:45:16.615021 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.614964    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-44.ec2.internal.18d5e1744eb7f519  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-44.ec2.internal,UID:ip-10-0-135-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-44.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-44.ec2.internal,},FirstTimestamp:2026-09-16 18:45:16.602774809 +0000 UTC m=+0.416363659,LastTimestamp:2026-09-16 18:45:16.602774809 +0000 UTC m=+0.416363659,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-44.ec2.internal,}"
Sep 16 18:45:16.618008 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.617988    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cmmnh"
Sep 16 18:45:16.620991 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.620939    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-44.ec2.internal.18d5e1744eb81985  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-44.ec2.internal,UID:ip-10-0-135-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-44.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-44.ec2.internal,},FirstTimestamp:2026-09-16 18:45:16.602784133 +0000 UTC m=+0.416372982,LastTimestamp:2026-09-16 18:45:16.602784133 +0000 UTC m=+0.416372982,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-44.ec2.internal,}"
Sep 16 18:45:16.626512 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.626451    2565 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-44.ec2.internal.18d5e1744eb7b408\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-44.ec2.internal.18d5e1744eb7b408  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-44.ec2.internal,UID:ip-10-0-135-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-44.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-44.ec2.internal,},FirstTimestamp:2026-09-16 18:45:16.602758152 +0000 UTC m=+0.416347001,LastTimestamp:2026-09-16 18:45:16.604133974 +0000 UTC m=+0.417722831,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-44.ec2.internal,}"
Sep 16 18:45:16.628248 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.628193    2565 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-44.ec2.internal.18d5e1744eb7f519\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-44.ec2.internal.18d5e1744eb7f519  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-44.ec2.internal,UID:ip-10-0-135-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-44.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-44.ec2.internal,},FirstTimestamp:2026-09-16 18:45:16.602774809 +0000 UTC m=+0.416363659,LastTimestamp:2026-09-16 18:45:16.604156999 +0000 UTC m=+0.417745864,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-44.ec2.internal,}"
Sep 16 18:45:16.629761 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.629744    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cmmnh"
Sep 16 18:45:16.634084 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.633566    2565 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:45:16.634084 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.633598    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:45:16.634084 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.633608    2565 server.go:87] "Starting device plugin registration server"
Sep 16 18:45:16.634084 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.633846    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:45:16.634084 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.633855    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:45:16.745072 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.634183    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:45:16.745072 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.634261    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:45:16.745072 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.634269    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:45:16.745072 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.634639    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:45:16.745072 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.634682    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:16.745072 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.637063    2565 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-44.ec2.internal.18d5e1744eb81985\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-44.ec2.internal.18d5e1744eb81985  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-44.ec2.internal,UID:ip-10-0-135-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-44.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-44.ec2.internal,},FirstTimestamp:2026-09-16 18:45:16.602784133 +0000 UTC m=+0.416372982,LastTimestamp:2026-09-16 18:45:16.604172005 +0000 UTC m=+0.417760861,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-44.ec2.internal,}"
Sep 16 18:45:16.745072 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.734642    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:16.745072 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.735392    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:16.745072 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.735418    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:16.745072 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.735427    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:16.745072 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.735446    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.745497 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.745394    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.745497 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.745408    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-44.ec2.internal\": node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:16.745772 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.745756    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:45:16.745810 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.745800    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:45:16.745842 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.745817    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:45:16.745875 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.745851    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:45:16.748771 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.748752    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:16.801365 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.801343    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:16.846210 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.846192    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-44.ec2.internal"]
Sep 16 18:45:16.846268 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.846253    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:16.847730 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.847713    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:16.847803 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.847740    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:16.847803 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.847750    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:16.850115 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.850090    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:16.850223 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.850209    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.850268 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.850237    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:16.850769 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.850749    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:16.850843 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.850792    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:16.850843 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.850812    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:16.851234 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.851216    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:16.851308 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.851250    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:16.851308 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.851267    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:16.853579 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.853566    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.853651 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.853590    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:16.854282 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.854268    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:16.854352 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.854293    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:16.854352 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.854302    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:16.879617 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.879597    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-44.ec2.internal\" not found" node="ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.883780 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.883766    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-44.ec2.internal\" not found" node="ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.885328 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.885313    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7722d781ffc02fceb2752ba72496be8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal\" (UID: \"7722d781ffc02fceb2752ba72496be8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.885396 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.885337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7722d781ffc02fceb2752ba72496be8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal\" (UID: \"7722d781ffc02fceb2752ba72496be8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.885396 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.885355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a0cc900755ac57a20096a2c2cbaeaf7-config\") pod \"kube-apiserver-proxy-ip-10-0-135-44.ec2.internal\" (UID: \"4a0cc900755ac57a20096a2c2cbaeaf7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.901828 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:16.901811    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:16.986424 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.986402    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7722d781ffc02fceb2752ba72496be8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal\" (UID: \"7722d781ffc02fceb2752ba72496be8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.986424 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.986423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7722d781ffc02fceb2752ba72496be8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal\" (UID: \"7722d781ffc02fceb2752ba72496be8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.986568 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.986438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a0cc900755ac57a20096a2c2cbaeaf7-config\") pod \"kube-apiserver-proxy-ip-10-0-135-44.ec2.internal\" (UID: \"4a0cc900755ac57a20096a2c2cbaeaf7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.986568 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.986463    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a0cc900755ac57a20096a2c2cbaeaf7-config\") pod \"kube-apiserver-proxy-ip-10-0-135-44.ec2.internal\" (UID: \"4a0cc900755ac57a20096a2c2cbaeaf7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.986568 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.986505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7722d781ffc02fceb2752ba72496be8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal\" (UID: \"7722d781ffc02fceb2752ba72496be8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:16.986568 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:16.986505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7722d781ffc02fceb2752ba72496be8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal\" (UID: \"7722d781ffc02fceb2752ba72496be8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:17.002086 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:17.002035    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:17.102875 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:17.102846    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:17.183004 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.182978    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:17.188371 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.188320    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:17.203823 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:17.203771    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:17.304354 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:17.304298    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:17.404777 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:17.404754    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:17.505257 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:17.505231    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:17.518396 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.518376    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:45:17.518521 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.518499    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:17.605399 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:17.605344    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:17.635767 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.635748    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:17.635846 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.635735    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:40:17 +0000 UTC" deadline="2028-03-18 06:17:15.52763795 +0000 UTC"
Sep 16 18:45:17.635916 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.635852    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13163h31m57.891790652s"
Sep 16 18:45:17.636574 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.636556    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:17.636696 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.636592    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:17.636696 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.636607    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:17.705488 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:17.705461    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:17.806438 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:17.806419    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:17.881553 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.881499    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:17.907475 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:17.907452    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-44.ec2.internal\" not found"
Sep 16 18:45:17.920073 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:17.920034    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7722d781ffc02fceb2752ba72496be8d.slice/crio-61e85dd1f9a706f31d5e1bc8a215f5b3a5413b5cadb3a9b89b466312b0e92a54 WatchSource:0}: Error finding container 61e85dd1f9a706f31d5e1bc8a215f5b3a5413b5cadb3a9b89b466312b0e92a54: Status 404 returned error can't find the container with id 61e85dd1f9a706f31d5e1bc8a215f5b3a5413b5cadb3a9b89b466312b0e92a54
Sep 16 18:45:17.920575 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:17.920560    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0cc900755ac57a20096a2c2cbaeaf7.slice/crio-22dcc665d9cdedae15d48f106c48a0270564a4bad37cf6c07ec5ec759ad928a6 WatchSource:0}: Error finding container 22dcc665d9cdedae15d48f106c48a0270564a4bad37cf6c07ec5ec759ad928a6: Status 404 returned error can't find the container with id 22dcc665d9cdedae15d48f106c48a0270564a4bad37cf6c07ec5ec759ad928a6
Sep 16 18:45:17.924660 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.924637    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:17.949513 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.949492    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:17.984277 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.984257    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:17.995894 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.995876    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:17.998005 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:17.997992    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal"
Sep 16 18:45:18.011707 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.011687    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:18.110714 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.110694    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:18.478430 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.478231    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:18.560994 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.560971    2565 apiserver.go:52] "Watching apiserver"
Sep 16 18:45:18.569540 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.569289    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:45:18.570887 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.570856    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-lfqsq","openshift-network-operator/iptables-alerter-j88k9","openshift-ovn-kubernetes/ovnkube-node-thbb5","kube-system/konnectivity-agent-j58qd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb","openshift-dns/node-resolver-9n9vk","openshift-image-registry/node-ca-phj72","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal","openshift-multus/multus-7kqt2","openshift-multus/network-metrics-daemon-qz4s2","kube-system/global-pull-secret-syncer-6kw62","kube-system/kube-apiserver-proxy-ip-10-0-135-44.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5qg59","openshift-multus/multus-additional-cni-plugins-4kq9p"]
Sep 16 18:45:18.573746 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.573722    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-phj72"
Sep 16 18:45:18.576567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.576513    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.576888 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.576865    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kq9zl\""
Sep 16 18:45:18.577381 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.577361    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:45:18.577481 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.577450    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:45:18.577629 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.577613    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:45:18.578672 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.578653    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.580344 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.580217    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:18.580549 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.580521    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:18.580776 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.580759    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6gmd7\""
Sep 16 18:45:18.581086 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.581066    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j58qd"
Sep 16 18:45:18.582407 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.582382    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:45:18.583932 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.583913    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.584684 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.584663    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:18.585414 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.585380    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:45:18.585588 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.585567    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:45:18.585818 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.585713    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zk2hq\""
Sep 16 18:45:18.585818 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.585745    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:45:18.586004 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.585975    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:45:18.586104 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.586085    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:45:18.586236 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.586218    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vjtvs\""
Sep 16 18:45:18.586670 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.585983    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:45:18.586670 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.586415    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:45:18.587841 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.587824    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9n9vk"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.590929    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593439    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/686c60bb-f175-4c94-9b73-129291c971b4-env-overrides\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/686c60bb-f175-4c94-9b73-129291c971b4-ovn-node-metrics-cert\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593509    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-socket-dir\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593587    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa789e9-6a45-4199-8137-895a502baed7-tmp-dir\") pod \"node-resolver-9n9vk\" (UID: \"8aa789e9-6a45-4199-8137-895a502baed7\") " pod="openshift-dns/node-resolver-9n9vk"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593627    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ae17fba-9a68-4849-a1d4-3c930a0daa61-host\") pod \"node-ca-phj72\" (UID: \"9ae17fba-9a68-4849-a1d4-3c930a0daa61\") " pod="openshift-image-registry/node-ca-phj72"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593700    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ae17fba-9a68-4849-a1d4-3c930a0daa61-serviceca\") pod \"node-ca-phj72\" (UID: \"9ae17fba-9a68-4849-a1d4-3c930a0daa61\") " pod="openshift-image-registry/node-ca-phj72"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593735    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-run\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593774    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7ktm\" (UniqueName: \"kubernetes.io/projected/3c8f259b-d7cb-4236-a384-474291299c70-kube-api-access-p7ktm\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593811    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-sysconfig\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-host\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bf127e9-9ab0-4581-a467-62bdaa8052f5-tmp\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.594013 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.593961    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594058    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-cni-bin\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/686c60bb-f175-4c94-9b73-129291c971b4-ovnkube-config\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:18.594094    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/686c60bb-f175-4c94-9b73-129291c971b4-ovnkube-script-lib\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594146    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-var-lib-kubelet\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594243    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-device-dir\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594313    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-sysctl-d\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-sysctl-conf\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-systemd\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594427    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-systemd-units\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-registration-dir\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8aa789e9-6a45-4199-8137-895a502baed7-hosts-file\") pod \"node-resolver-9n9vk\" (UID: \"8aa789e9-6a45-4199-8137-895a502baed7\") " pod="openshift-dns/node-resolver-9n9vk"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7xt\" (UniqueName: \"kubernetes.io/projected/8aa789e9-6a45-4199-8137-895a502baed7-kube-api-access-8r7xt\") pod \"node-resolver-9n9vk\" (UID: \"8aa789e9-6a45-4199-8137-895a502baed7\") " pod="openshift-dns/node-resolver-9n9vk"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-lib-modules\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.594828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-slash\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.595760 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-run-netns\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.595760 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594750    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-var-lib-openvswitch\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.595760 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-cni-netd\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.595760 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594813    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d00f6df-66e0-472a-abfc-9fd5e349ef69-konnectivity-ca\") pod \"konnectivity-agent-j58qd\" (UID: \"6d00f6df-66e0-472a-abfc-9fd5e349ef69\") " pod="kube-system/konnectivity-agent-j58qd"
Sep 16 18:45:18.595760 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594851    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-modprobe-d\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.595760 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.594957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-kubernetes\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.596068 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.595792    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5kv\" (UniqueName: \"kubernetes.io/projected/1bf127e9-9ab0-4581-a467-62bdaa8052f5-kube-api-access-nl5kv\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.596068 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.595821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-kubelet\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.596068 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.595845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-run-openvswitch\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.596068 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.595869    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-run-ovn\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.596068 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.595891    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-node-log\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.596068 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.595913    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfx5l\" (UniqueName: \"kubernetes.io/projected/9ae17fba-9a68-4849-a1d4-3c930a0daa61-kube-api-access-zfx5l\") pod \"node-ca-phj72\" (UID: \"9ae17fba-9a68-4849-a1d4-3c930a0daa61\") " pod="openshift-image-registry/node-ca-phj72"
Sep 16 18:45:18.596439 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.596100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-sys\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.596439 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.596141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-tuned\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.596439 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.596165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-log-socket\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.596439 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.596189    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwtjn\" (UniqueName: \"kubernetes.io/projected/686c60bb-f175-4c94-9b73-129291c971b4-kube-api-access-qwtjn\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.596439 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.596213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d00f6df-66e0-472a-abfc-9fd5e349ef69-agent-certs\") pod \"konnectivity-agent-j58qd\" (UID: \"6d00f6df-66e0-472a-abfc-9fd5e349ef69\") " pod="kube-system/konnectivity-agent-j58qd"
Sep 16 18:45:18.596439 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.596233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.596439 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.596253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-sys-fs\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.596439 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.596288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-run-systemd\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.596439 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.596327    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-etc-openvswitch\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.597451 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.597432    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j88k9"
Sep 16 18:45:18.598097 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.598078    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:45:18.598527 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.598509    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:45:18.598792 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.598773    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:45:18.598861 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.598823    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-b4qcn\""
Sep 16 18:45:18.598861 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.598773    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:45:18.599019 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.598999    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-88f4q\""
Sep 16 18:45:18.599171 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.599156    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:45:18.600878 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.600856    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:18.600966 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:18.600932    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:18.601966 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.601946    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:18.602043 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.602016    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:45:18.602127 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.602047    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:45:18.602226 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.602175    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-74qwp\""
Sep 16 18:45:18.602324 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.602309    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h6nrr\""
Sep 16 18:45:18.602681 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.602660    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:45:18.602756 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.602739    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:18.602997 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.602979    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:45:18.603759 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.603742    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:18.603853 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:18.603800    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:18.606385 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.606363    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.609030 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.609014    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:45:18.609121 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.609034    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:45:18.609335 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.609319    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jgp7k\""
Sep 16 18:45:18.622877 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.622856    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:18.666437 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.666418    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-72nqt"
Sep 16 18:45:18.679404 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.679382    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-72nqt"
Sep 16 18:45:18.684716 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.684688    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:45:18.696545 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:18.696662 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-run-systemd\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.696662 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ae17fba-9a68-4849-a1d4-3c930a0daa61-serviceca\") pod \"node-ca-phj72\" (UID: \"9ae17fba-9a68-4849-a1d4-3c930a0daa61\") " pod="openshift-image-registry/node-ca-phj72"
Sep 16 18:45:18.696662 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-var-lib-cni-bin\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.696662 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696644    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:18.696880 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-sysconfig\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.696880 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bf127e9-9ab0-4581-a467-62bdaa8052f5-tmp\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.696880 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696738    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.696880 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-cni-bin\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.696880 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/686c60bb-f175-4c94-9b73-129291c971b4-ovnkube-script-lib\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.696880 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-device-dir\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.696880 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-var-lib-kubelet\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696888    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-run-systemd\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4664j\" (UniqueName: \"kubernetes.io/projected/66a5cb54-a1df-40ca-a945-a5378898cf93-kube-api-access-4664j\") pod \"iptables-alerter-j88k9\" (UID: \"66a5cb54-a1df-40ca-a945-a5378898cf93\") " pod="openshift-network-operator/iptables-alerter-j88k9"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.696976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-systemd-units\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697047    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-hostroot\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66a5cb54-a1df-40ca-a945-a5378898cf93-host-slash\") pod \"iptables-alerter-j88k9\" (UID: \"66a5cb54-a1df-40ca-a945-a5378898cf93\") " pod="openshift-network-operator/iptables-alerter-j88k9"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697141    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-device-dir\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-lib-modules\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-slash\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697250    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-sysconfig\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697260    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-lib-modules\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-run-netns\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-var-lib-kubelet\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697331    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-var-lib-openvswitch\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-systemd-units\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697458    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-cni-netd\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ae17fba-9a68-4849-a1d4-3c930a0daa61-serviceca\") pod \"node-ca-phj72\" (UID: \"9ae17fba-9a68-4849-a1d4-3c930a0daa61\") " pod="openshift-image-registry/node-ca-phj72"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697480    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-slash\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697513    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-socket-dir\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697536    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-cni-netd\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5kv\" (UniqueName: \"kubernetes.io/projected/1bf127e9-9ab0-4581-a467-62bdaa8052f5-kube-api-access-nl5kv\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-run-netns\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-kubelet\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-run-openvswitch\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-run-ovn\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-cni-bin\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697648    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-socket-dir\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697648    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-var-lib-openvswitch\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697712    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-node-log\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-node-log\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-log-socket\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.699217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697778    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-run-ovn\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7ktm\" (UniqueName: \"kubernetes.io/projected/3c8f259b-d7cb-4236-a384-474291299c70-kube-api-access-p7ktm\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-host-kubelet\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-run-openvswitch\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697858    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-multus-conf-dir\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-etc-kubernetes\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697915    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/686c60bb-f175-4c94-9b73-129291c971b4-ovnkube-script-lib\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.697928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjpj\" (UniqueName: \"kubernetes.io/projected/4ad64d2a-fd0c-45f9-be97-119db12f0292-kube-api-access-qnjpj\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698006    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/66a5cb54-a1df-40ca-a945-a5378898cf93-iptables-alerter-script\") pod \"iptables-alerter-j88k9\" (UID: \"66a5cb54-a1df-40ca-a945-a5378898cf93\") " pod="openshift-network-operator/iptables-alerter-j88k9"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-log-socket\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698072    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28a43599-1a6c-4f0e-80a9-032b45196140-dbus\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698130    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-cnibin\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-etc-openvswitch\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/686c60bb-f175-4c94-9b73-129291c971b4-env-overrides\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698329    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686c60bb-f175-4c94-9b73-129291c971b4-etc-openvswitch\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/686c60bb-f175-4c94-9b73-129291c971b4-ovn-node-metrics-cert\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.700156 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698398    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa789e9-6a45-4199-8137-895a502baed7-tmp-dir\") pod \"node-resolver-9n9vk\" (UID: \"8aa789e9-6a45-4199-8137-895a502baed7\") " pod="openshift-dns/node-resolver-9n9vk"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ae17fba-9a68-4849-a1d4-3c930a0daa61-host\") pod \"node-ca-phj72\" (UID: \"9ae17fba-9a68-4849-a1d4-3c930a0daa61\") " pod="openshift-image-registry/node-ca-phj72"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-run\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-var-lib-cni-multus\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmftq\" (UniqueName: \"kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq\") pod \"network-check-target-lfqsq\" (UID: \"9f5c39f3-f783-4d49-b3f1-3cd4392d347c\") " pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698628    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-host\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698659    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/686c60bb-f175-4c94-9b73-129291c971b4-env-overrides\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/686c60bb-f175-4c94-9b73-129291c971b4-ovnkube-config\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698744    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-system-cni-dir\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698768    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd9n9\" (UniqueName: \"kubernetes.io/projected/5dda191a-363f-44c4-9f13-d1f59108abbc-kube-api-access-bd9n9\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698789    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28a43599-1a6c-4f0e-80a9-032b45196140-kubelet-config\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-run\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-var-lib-kubelet\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698912    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npwx2\" (UniqueName: \"kubernetes.io/projected/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-kube-api-access-npwx2\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698949    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-sysctl-d\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.701012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.698977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-sysctl-conf\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-systemd\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-registration-dir\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699057    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ad64d2a-fd0c-45f9-be97-119db12f0292-cni-binary-copy\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-run-k8s-cni-cncf-io\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa789e9-6a45-4199-8137-895a502baed7-tmp-dir\") pod \"node-resolver-9n9vk\" (UID: \"8aa789e9-6a45-4199-8137-895a502baed7\") " pod="openshift-dns/node-resolver-9n9vk"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699211    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-host\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4ad64d2a-fd0c-45f9-be97-119db12f0292-multus-daemon-config\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ae17fba-9a68-4849-a1d4-3c930a0daa61-host\") pod \"node-ca-phj72\" (UID: \"9ae17fba-9a68-4849-a1d4-3c930a0daa61\") " pod="openshift-image-registry/node-ca-phj72"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699259    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-run-multus-certs\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8aa789e9-6a45-4199-8137-895a502baed7-hosts-file\") pod \"node-resolver-9n9vk\" (UID: \"8aa789e9-6a45-4199-8137-895a502baed7\") " pod="openshift-dns/node-resolver-9n9vk"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7xt\" (UniqueName: \"kubernetes.io/projected/8aa789e9-6a45-4199-8137-895a502baed7-kube-api-access-8r7xt\") pod \"node-resolver-9n9vk\" (UID: \"8aa789e9-6a45-4199-8137-895a502baed7\") " pod="openshift-dns/node-resolver-9n9vk"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d00f6df-66e0-472a-abfc-9fd5e349ef69-konnectivity-ca\") pod \"konnectivity-agent-j58qd\" (UID: \"6d00f6df-66e0-472a-abfc-9fd5e349ef69\") " pod="kube-system/konnectivity-agent-j58qd"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-multus-cni-dir\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-cnibin\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699447    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-multus-socket-dir-parent\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699472    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-run-netns\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.701954 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-system-cni-dir\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-modprobe-d\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699599    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-kubernetes\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699750    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-sys-fs\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699782    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/686c60bb-f175-4c94-9b73-129291c971b4-ovnkube-config\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699786    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699837    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfx5l\" (UniqueName: \"kubernetes.io/projected/9ae17fba-9a68-4849-a1d4-3c930a0daa61-kube-api-access-zfx5l\") pod \"node-ca-phj72\" (UID: \"9ae17fba-9a68-4849-a1d4-3c930a0daa61\") " pod="openshift-image-registry/node-ca-phj72"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699864    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-kubernetes\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699899    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-sys\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699928    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8aa789e9-6a45-4199-8137-895a502baed7-hosts-file\") pod \"node-resolver-9n9vk\" (UID: \"8aa789e9-6a45-4199-8137-895a502baed7\") " pod="openshift-dns/node-resolver-9n9vk"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-tuned\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.699992    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwtjn\" (UniqueName: \"kubernetes.io/projected/686c60bb-f175-4c94-9b73-129291c971b4-kube-api-access-qwtjn\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.700041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d00f6df-66e0-472a-abfc-9fd5e349ef69-agent-certs\") pod \"konnectivity-agent-j58qd\" (UID: \"6d00f6df-66e0-472a-abfc-9fd5e349ef69\") " pod="kube-system/konnectivity-agent-j58qd"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.700071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.700194    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.700244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-modprobe-d\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.702851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.700304    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-sys-fs\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.703723 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.700351    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-registration-dir\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.703723 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.700455    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-sysctl-d\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.703723 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.700552    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-sysctl-conf\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.703723 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.700606    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-systemd\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.703723 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.700789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d00f6df-66e0-472a-abfc-9fd5e349ef69-konnectivity-ca\") pod \"konnectivity-agent-j58qd\" (UID: \"6d00f6df-66e0-472a-abfc-9fd5e349ef69\") " pod="kube-system/konnectivity-agent-j58qd"
Sep 16 18:45:18.703723 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.700901    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bf127e9-9ab0-4581-a467-62bdaa8052f5-sys\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.703723 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.700948    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c8f259b-d7cb-4236-a384-474291299c70-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.703723 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.702695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bf127e9-9ab0-4581-a467-62bdaa8052f5-tmp\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.704653 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.704563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/686c60bb-f175-4c94-9b73-129291c971b4-ovn-node-metrics-cert\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.704653 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.704608    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d00f6df-66e0-472a-abfc-9fd5e349ef69-agent-certs\") pod \"konnectivity-agent-j58qd\" (UID: \"6d00f6df-66e0-472a-abfc-9fd5e349ef69\") " pod="kube-system/konnectivity-agent-j58qd"
Sep 16 18:45:18.704653 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.704568    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1bf127e9-9ab0-4581-a467-62bdaa8052f5-etc-tuned\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.719767 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.719733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7ktm\" (UniqueName: \"kubernetes.io/projected/3c8f259b-d7cb-4236-a384-474291299c70-kube-api-access-p7ktm\") pod \"aws-ebs-csi-driver-node-mgvpb\" (UID: \"3c8f259b-d7cb-4236-a384-474291299c70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.719905 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.719879    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5kv\" (UniqueName: \"kubernetes.io/projected/1bf127e9-9ab0-4581-a467-62bdaa8052f5-kube-api-access-nl5kv\") pod \"tuned-5qg59\" (UID: \"1bf127e9-9ab0-4581-a467-62bdaa8052f5\") " pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.720882 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.720850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfx5l\" (UniqueName: \"kubernetes.io/projected/9ae17fba-9a68-4849-a1d4-3c930a0daa61-kube-api-access-zfx5l\") pod \"node-ca-phj72\" (UID: \"9ae17fba-9a68-4849-a1d4-3c930a0daa61\") " pod="openshift-image-registry/node-ca-phj72"
Sep 16 18:45:18.721931 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.721909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwtjn\" (UniqueName: \"kubernetes.io/projected/686c60bb-f175-4c94-9b73-129291c971b4-kube-api-access-qwtjn\") pod \"ovnkube-node-thbb5\" (UID: \"686c60bb-f175-4c94-9b73-129291c971b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.724099 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.724060    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7xt\" (UniqueName: \"kubernetes.io/projected/8aa789e9-6a45-4199-8137-895a502baed7-kube-api-access-8r7xt\") pod \"node-resolver-9n9vk\" (UID: \"8aa789e9-6a45-4199-8137-895a502baed7\") " pod="openshift-dns/node-resolver-9n9vk"
Sep 16 18:45:18.759019 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.758940    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-44.ec2.internal" event={"ID":"4a0cc900755ac57a20096a2c2cbaeaf7","Type":"ContainerStarted","Data":"22dcc665d9cdedae15d48f106c48a0270564a4bad37cf6c07ec5ec759ad928a6"}
Sep 16 18:45:18.760622 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.760594    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal" event={"ID":"7722d781ffc02fceb2752ba72496be8d","Type":"ContainerStarted","Data":"61e85dd1f9a706f31d5e1bc8a215f5b3a5413b5cadb3a9b89b466312b0e92a54"}
Sep 16 18:45:18.800891 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.800864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4664j\" (UniqueName: \"kubernetes.io/projected/66a5cb54-a1df-40ca-a945-a5378898cf93-kube-api-access-4664j\") pod \"iptables-alerter-j88k9\" (UID: \"66a5cb54-a1df-40ca-a945-a5378898cf93\") " pod="openshift-network-operator/iptables-alerter-j88k9"
Sep 16 18:45:18.801007 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.800905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.801007 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.800932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-hostroot\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801007 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.800955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66a5cb54-a1df-40ca-a945-a5378898cf93-host-slash\") pod \"iptables-alerter-j88k9\" (UID: \"66a5cb54-a1df-40ca-a945-a5378898cf93\") " pod="openshift-network-operator/iptables-alerter-j88k9"
Sep 16 18:45:18.801007 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.800986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-multus-conf-dir\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801244 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-etc-kubernetes\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801244 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjpj\" (UniqueName: \"kubernetes.io/projected/4ad64d2a-fd0c-45f9-be97-119db12f0292-kube-api-access-qnjpj\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801244 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/66a5cb54-a1df-40ca-a945-a5378898cf93-iptables-alerter-script\") pod \"iptables-alerter-j88k9\" (UID: \"66a5cb54-a1df-40ca-a945-a5378898cf93\") " pod="openshift-network-operator/iptables-alerter-j88k9"
Sep 16 18:45:18.801244 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28a43599-1a6c-4f0e-80a9-032b45196140-dbus\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:18.801244 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-cnibin\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.801244 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-var-lib-cni-multus\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801244 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmftq\" (UniqueName: \"kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq\") pod \"network-check-target-lfqsq\" (UID: \"9f5c39f3-f783-4d49-b3f1-3cd4392d347c\") " pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:18.801244 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-system-cni-dir\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd9n9\" (UniqueName: \"kubernetes.io/projected/5dda191a-363f-44c4-9f13-d1f59108abbc-kube-api-access-bd9n9\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28a43599-1a6c-4f0e-80a9-032b45196140-kubelet-config\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801332    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-var-lib-kubelet\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npwx2\" (UniqueName: \"kubernetes.io/projected/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-kube-api-access-npwx2\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ad64d2a-fd0c-45f9-be97-119db12f0292-cni-binary-copy\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801410    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-run-k8s-cni-cncf-io\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801435    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4ad64d2a-fd0c-45f9-be97-119db12f0292-multus-daemon-config\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-run-multus-certs\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-multus-cni-dir\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-cnibin\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-multus-socket-dir-parent\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-run-netns\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-system-cni-dir\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.801635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801621    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.802363 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.802363 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:18.802363 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-var-lib-cni-bin\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.802363 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.801734    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:18.802363 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:18.801865    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:18.802363 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:18.801940    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret podName:28a43599-1a6c-4f0e-80a9-032b45196140 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:19.301907863 +0000 UTC m=+3.115496723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret") pod "global-pull-secret-syncer-6kw62" (UID: "28a43599-1a6c-4f0e-80a9-032b45196140") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.802735    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-var-lib-kubelet\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.802795    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-hostroot\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.802810    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.802836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66a5cb54-a1df-40ca-a945-a5378898cf93-host-slash\") pod \"iptables-alerter-j88k9\" (UID: \"66a5cb54-a1df-40ca-a945-a5378898cf93\") " pod="openshift-network-operator/iptables-alerter-j88k9"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.802876    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-multus-conf-dir\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.802913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-etc-kubernetes\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-cnibin\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28a43599-1a6c-4f0e-80a9-032b45196140-dbus\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803104    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-run-k8s-cni-cncf-io\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28a43599-1a6c-4f0e-80a9-032b45196140-kubelet-config\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-var-lib-cni-multus\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-run-netns\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:18.803281    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803333    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-multus-socket-dir-parent\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:18.803351    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs podName:5dda191a-363f-44c4-9f13-d1f59108abbc nodeName:}" failed. No retries permitted until 2026-09-16 18:45:19.303331982 +0000 UTC m=+3.116920821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs") pod "network-metrics-daemon-qz4s2" (UID: "5dda191a-363f-44c4-9f13-d1f59108abbc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803389    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-run-multus-certs\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-host-var-lib-cni-bin\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.803633 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803420    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ad64d2a-fd0c-45f9-be97-119db12f0292-cni-binary-copy\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.804418 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.804418 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-system-cni-dir\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.804418 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-system-cni-dir\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.804418 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803513    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-multus-cni-dir\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.804418 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803564    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ad64d2a-fd0c-45f9-be97-119db12f0292-cnibin\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.804418 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4ad64d2a-fd0c-45f9-be97-119db12f0292-multus-daemon-config\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.804418 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803691    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/66a5cb54-a1df-40ca-a945-a5378898cf93-iptables-alerter-script\") pod \"iptables-alerter-j88k9\" (UID: \"66a5cb54-a1df-40ca-a945-a5378898cf93\") " pod="openshift-network-operator/iptables-alerter-j88k9"
Sep 16 18:45:18.804418 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.803967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.804418 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.804059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.814209 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.814169    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4664j\" (UniqueName: \"kubernetes.io/projected/66a5cb54-a1df-40ca-a945-a5378898cf93-kube-api-access-4664j\") pod \"iptables-alerter-j88k9\" (UID: \"66a5cb54-a1df-40ca-a945-a5378898cf93\") " pod="openshift-network-operator/iptables-alerter-j88k9"
Sep 16 18:45:18.814775 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:18.814586    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:18.814775 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:18.814610    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:18.814775 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:18.814623    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rmftq for pod openshift-network-diagnostics/network-check-target-lfqsq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:18.814775 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:18.814709    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq podName:9f5c39f3-f783-4d49-b3f1-3cd4392d347c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:19.314692321 +0000 UTC m=+3.128281176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rmftq" (UniqueName: "kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq") pod "network-check-target-lfqsq" (UID: "9f5c39f3-f783-4d49-b3f1-3cd4392d347c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:18.817091 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.817067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjpj\" (UniqueName: \"kubernetes.io/projected/4ad64d2a-fd0c-45f9-be97-119db12f0292-kube-api-access-qnjpj\") pod \"multus-7kqt2\" (UID: \"4ad64d2a-fd0c-45f9-be97-119db12f0292\") " pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.817465 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.817415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npwx2\" (UniqueName: \"kubernetes.io/projected/023ad12c-f4a4-43cf-b91a-7004d2eff4ed-kube-api-access-npwx2\") pod \"multus-additional-cni-plugins-4kq9p\" (UID: \"023ad12c-f4a4-43cf-b91a-7004d2eff4ed\") " pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:18.819233 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.819212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd9n9\" (UniqueName: \"kubernetes.io/projected/5dda191a-363f-44c4-9f13-d1f59108abbc-kube-api-access-bd9n9\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:18.891782 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.891756    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-phj72"
Sep 16 18:45:18.904380 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.904355    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5qg59"
Sep 16 18:45:18.917537 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.917515    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:18.923433 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.923417    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j58qd"
Sep 16 18:45:18.931403 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.931382    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb"
Sep 16 18:45:18.940276 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.940253    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9n9vk"
Sep 16 18:45:18.948376 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.948353    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7kqt2"
Sep 16 18:45:18.958188 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.958169    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j88k9"
Sep 16 18:45:18.963865 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:18.963844    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4kq9p"
Sep 16 18:45:19.305580 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:19.305542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:19.305764 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:19.305595    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:19.305764 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:19.305668    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:19.305764 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:19.305735    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs podName:5dda191a-363f-44c4-9f13-d1f59108abbc nodeName:}" failed. No retries permitted until 2026-09-16 18:45:20.305713606 +0000 UTC m=+4.119302446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs") pod "network-metrics-daemon-qz4s2" (UID: "5dda191a-363f-44c4-9f13-d1f59108abbc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:19.305764 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:19.305745    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:19.306004 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:19.305791    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret podName:28a43599-1a6c-4f0e-80a9-032b45196140 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:20.305780353 +0000 UTC m=+4.119369197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret") pod "global-pull-secret-syncer-6kw62" (UID: "28a43599-1a6c-4f0e-80a9-032b45196140") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:19.406641 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:19.406610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmftq\" (UniqueName: \"kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq\") pod \"network-check-target-lfqsq\" (UID: \"9f5c39f3-f783-4d49-b3f1-3cd4392d347c\") " pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:19.406791 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:19.406775    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:19.406833 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:19.406798    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:19.406833 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:19.406811    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rmftq for pod openshift-network-diagnostics/network-check-target-lfqsq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:19.406915 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:19.406867    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq podName:9f5c39f3-f783-4d49-b3f1-3cd4392d347c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:20.406849573 +0000 UTC m=+4.220438417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rmftq" (UniqueName: "kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq") pod "network-check-target-lfqsq" (UID: "9f5c39f3-f783-4d49-b3f1-3cd4392d347c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:19.683407 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:19.683316    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:19 +0000 UTC" deadline="2028-05-06 05:19:33.14108857 +0000 UTC"
Sep 16 18:45:19.683407 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:19.683356    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14338h34m13.457738038s"
Sep 16 18:45:19.925418 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:19.925385    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad64d2a_fd0c_45f9_be97_119db12f0292.slice/crio-9b051b2cecc164fc6f855b9b4f3f713978e0c4bd387edbc578403eaaf6df7476 WatchSource:0}: Error finding container 9b051b2cecc164fc6f855b9b4f3f713978e0c4bd387edbc578403eaaf6df7476: Status 404 returned error can't find the container with id 9b051b2cecc164fc6f855b9b4f3f713978e0c4bd387edbc578403eaaf6df7476
Sep 16 18:45:19.927509 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:19.927483    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d00f6df_66e0_472a_abfc_9fd5e349ef69.slice/crio-f0ade9648e571fd9394dd1e72c77c19715fd31d86f57697a29d87b7499d0a4c9 WatchSource:0}: Error finding container f0ade9648e571fd9394dd1e72c77c19715fd31d86f57697a29d87b7499d0a4c9: Status 404 returned error can't find the container with id f0ade9648e571fd9394dd1e72c77c19715fd31d86f57697a29d87b7499d0a4c9
Sep 16 18:45:19.951643 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:19.951617    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa789e9_6a45_4199_8137_895a502baed7.slice/crio-53afc2c9777845be0fb54b315f045ef0a4f778c771b717038205775f755a0daa WatchSource:0}: Error finding container 53afc2c9777845be0fb54b315f045ef0a4f778c771b717038205775f755a0daa: Status 404 returned error can't find the container with id 53afc2c9777845be0fb54b315f045ef0a4f778c771b717038205775f755a0daa
Sep 16 18:45:19.955685 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:19.955663    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686c60bb_f175_4c94_9b73_129291c971b4.slice/crio-af9fc06860dfd0a3966c123677766213852b8825d1a7af569f5e3af74efd61e9 WatchSource:0}: Error finding container af9fc06860dfd0a3966c123677766213852b8825d1a7af569f5e3af74efd61e9: Status 404 returned error can't find the container with id af9fc06860dfd0a3966c123677766213852b8825d1a7af569f5e3af74efd61e9
Sep 16 18:45:19.957100 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:19.957040    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8f259b_d7cb_4236_a384_474291299c70.slice/crio-9cddb386b3f67a178a4cec21543efd5f5983b4c8082329c95e15fa3eeca85715 WatchSource:0}: Error finding container 9cddb386b3f67a178a4cec21543efd5f5983b4c8082329c95e15fa3eeca85715: Status 404 returned error can't find the container with id 9cddb386b3f67a178a4cec21543efd5f5983b4c8082329c95e15fa3eeca85715
Sep 16 18:45:19.958307 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:19.958286    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023ad12c_f4a4_43cf_b91a_7004d2eff4ed.slice/crio-7b7105eb03d67fafdd336639fb470490d91fb71a6e6a427ce13ec0088ef83ce2 WatchSource:0}: Error finding container 7b7105eb03d67fafdd336639fb470490d91fb71a6e6a427ce13ec0088ef83ce2: Status 404 returned error can't find the container with id 7b7105eb03d67fafdd336639fb470490d91fb71a6e6a427ce13ec0088ef83ce2
Sep 16 18:45:19.958985 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:19.958965    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a5cb54_a1df_40ca_a945_a5378898cf93.slice/crio-8bd2184890a3bfe26f7d4a0606f16df860382c611ea06d6dee1e45be868edb20 WatchSource:0}: Error finding container 8bd2184890a3bfe26f7d4a0606f16df860382c611ea06d6dee1e45be868edb20: Status 404 returned error can't find the container with id 8bd2184890a3bfe26f7d4a0606f16df860382c611ea06d6dee1e45be868edb20
Sep 16 18:45:19.961108 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:19.960828    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae17fba_9a68_4849_a1d4_3c930a0daa61.slice/crio-38642ee53eb2e20469c3aff113acf1be6ab21a80a65fbd36fee90446fe905de8 WatchSource:0}: Error finding container 38642ee53eb2e20469c3aff113acf1be6ab21a80a65fbd36fee90446fe905de8: Status 404 returned error can't find the container with id 38642ee53eb2e20469c3aff113acf1be6ab21a80a65fbd36fee90446fe905de8
Sep 16 18:45:19.961492 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:19.961466    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf127e9_9ab0_4581_a467_62bdaa8052f5.slice/crio-ebb4c73fccb4746505403445830e2afdaa05056a93a7fc8c74ba81c8a7806f54 WatchSource:0}: Error finding container ebb4c73fccb4746505403445830e2afdaa05056a93a7fc8c74ba81c8a7806f54: Status 404 returned error can't find the container with id ebb4c73fccb4746505403445830e2afdaa05056a93a7fc8c74ba81c8a7806f54
Sep 16 18:45:20.312915 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.312890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:20.313023 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.312927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:20.313083 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:20.313048    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:20.313083 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:20.313057    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:20.313251 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:20.313108    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret podName:28a43599-1a6c-4f0e-80a9-032b45196140 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:22.313088467 +0000 UTC m=+6.126677321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret") pod "global-pull-secret-syncer-6kw62" (UID: "28a43599-1a6c-4f0e-80a9-032b45196140") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:20.313251 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:20.313152    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs podName:5dda191a-363f-44c4-9f13-d1f59108abbc nodeName:}" failed. No retries permitted until 2026-09-16 18:45:22.313136902 +0000 UTC m=+6.126725745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs") pod "network-metrics-daemon-qz4s2" (UID: "5dda191a-363f-44c4-9f13-d1f59108abbc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:20.413755 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.413721    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmftq\" (UniqueName: \"kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq\") pod \"network-check-target-lfqsq\" (UID: \"9f5c39f3-f783-4d49-b3f1-3cd4392d347c\") " pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:20.413975 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:20.413957    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:20.414060 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:20.413982    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:20.414060 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:20.413995    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rmftq for pod openshift-network-diagnostics/network-check-target-lfqsq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:20.414060 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:20.414053    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq podName:9f5c39f3-f783-4d49-b3f1-3cd4392d347c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:22.414034553 +0000 UTC m=+6.227623398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rmftq" (UniqueName: "kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq") pod "network-check-target-lfqsq" (UID: "9f5c39f3-f783-4d49-b3f1-3cd4392d347c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:20.683608 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.683501    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:19 +0000 UTC" deadline="2028-06-21 15:03:38.241933382 +0000 UTC"
Sep 16 18:45:20.683608 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.683562    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15452h18m17.558375437s"
Sep 16 18:45:20.747813 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.747269    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:20.747813 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.747308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:20.747813 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:20.747410    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:20.747813 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.747481    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:20.747813 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:20.747606    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:20.747813 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:20.747693    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:20.769403 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.769334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-phj72" event={"ID":"9ae17fba-9a68-4849-a1d4-3c930a0daa61","Type":"ContainerStarted","Data":"38642ee53eb2e20469c3aff113acf1be6ab21a80a65fbd36fee90446fe905de8"}
Sep 16 18:45:20.772224 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.772158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j88k9" event={"ID":"66a5cb54-a1df-40ca-a945-a5378898cf93","Type":"ContainerStarted","Data":"8bd2184890a3bfe26f7d4a0606f16df860382c611ea06d6dee1e45be868edb20"}
Sep 16 18:45:20.774080 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.774007    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb" event={"ID":"3c8f259b-d7cb-4236-a384-474291299c70","Type":"ContainerStarted","Data":"9cddb386b3f67a178a4cec21543efd5f5983b4c8082329c95e15fa3eeca85715"}
Sep 16 18:45:20.780828 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.780803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9n9vk" event={"ID":"8aa789e9-6a45-4199-8137-895a502baed7","Type":"ContainerStarted","Data":"53afc2c9777845be0fb54b315f045ef0a4f778c771b717038205775f755a0daa"}
Sep 16 18:45:20.799253 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.799222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j58qd" event={"ID":"6d00f6df-66e0-472a-abfc-9fd5e349ef69","Type":"ContainerStarted","Data":"f0ade9648e571fd9394dd1e72c77c19715fd31d86f57697a29d87b7499d0a4c9"}
Sep 16 18:45:20.806087 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.805949    2565 generic.go:356] "Generic (PLEG): container finished" podID="7722d781ffc02fceb2752ba72496be8d" containerID="d1b63b07907949c1acd0a6d2ddd5cfdcbe4e7689a1fd6596bfe6681710af5f12" exitCode=0
Sep 16 18:45:20.806087 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.806043    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal" event={"ID":"7722d781ffc02fceb2752ba72496be8d","Type":"ContainerDied","Data":"d1b63b07907949c1acd0a6d2ddd5cfdcbe4e7689a1fd6596bfe6681710af5f12"}
Sep 16 18:45:20.827543 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.827519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5qg59" event={"ID":"1bf127e9-9ab0-4581-a467-62bdaa8052f5","Type":"ContainerStarted","Data":"ebb4c73fccb4746505403445830e2afdaa05056a93a7fc8c74ba81c8a7806f54"}
Sep 16 18:45:20.838326 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.837933    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kq9p" event={"ID":"023ad12c-f4a4-43cf-b91a-7004d2eff4ed","Type":"ContainerStarted","Data":"7b7105eb03d67fafdd336639fb470490d91fb71a6e6a427ce13ec0088ef83ce2"}
Sep 16 18:45:20.851559 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.851511    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5" event={"ID":"686c60bb-f175-4c94-9b73-129291c971b4","Type":"ContainerStarted","Data":"af9fc06860dfd0a3966c123677766213852b8825d1a7af569f5e3af74efd61e9"}
Sep 16 18:45:20.861256 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.861208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7kqt2" event={"ID":"4ad64d2a-fd0c-45f9-be97-119db12f0292","Type":"ContainerStarted","Data":"9b051b2cecc164fc6f855b9b4f3f713978e0c4bd387edbc578403eaaf6df7476"}
Sep 16 18:45:20.880865 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:20.880240    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-44.ec2.internal" event={"ID":"4a0cc900755ac57a20096a2c2cbaeaf7","Type":"ContainerStarted","Data":"f7b76e10ad89d7c553a7a363397ef915022b36330a2f77be9716a31b9e633848"}
Sep 16 18:45:21.898847 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:21.898765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal" event={"ID":"7722d781ffc02fceb2752ba72496be8d","Type":"ContainerStarted","Data":"1be662a55779aefbc128b88c195a3ecbf67fe80a67d9ecc33c464eb32c3e5591"}
Sep 16 18:45:21.916413 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:21.915193    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-44.ec2.internal" podStartSLOduration=3.915170322 podStartE2EDuration="3.915170322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:20.896106547 +0000 UTC m=+4.709695410" watchObservedRunningTime="2026-09-16 18:45:21.915170322 +0000 UTC m=+5.728759190"
Sep 16 18:45:22.330950 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:22.330238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:22.330950 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:22.330481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:22.330950 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:22.330504    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:22.330950 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:22.330579    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs podName:5dda191a-363f-44c4-9f13-d1f59108abbc nodeName:}" failed. No retries permitted until 2026-09-16 18:45:26.330558968 +0000 UTC m=+10.144147832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs") pod "network-metrics-daemon-qz4s2" (UID: "5dda191a-363f-44c4-9f13-d1f59108abbc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:22.330950 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:22.330632    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:22.330950 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:22.330686    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret podName:28a43599-1a6c-4f0e-80a9-032b45196140 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:26.330668908 +0000 UTC m=+10.144257751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret") pod "global-pull-secret-syncer-6kw62" (UID: "28a43599-1a6c-4f0e-80a9-032b45196140") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:22.431474 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:22.431435    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmftq\" (UniqueName: \"kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq\") pod \"network-check-target-lfqsq\" (UID: \"9f5c39f3-f783-4d49-b3f1-3cd4392d347c\") " pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:22.431630 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:22.431604    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:22.431630 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:22.431625    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:22.431749 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:22.431640    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rmftq for pod openshift-network-diagnostics/network-check-target-lfqsq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:22.431749 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:22.431704    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq podName:9f5c39f3-f783-4d49-b3f1-3cd4392d347c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:26.431682562 +0000 UTC m=+10.245271417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rmftq" (UniqueName: "kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq") pod "network-check-target-lfqsq" (UID: "9f5c39f3-f783-4d49-b3f1-3cd4392d347c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:22.749874 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:22.749799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:22.750038 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:22.749943    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:22.750728 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:22.750338    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:22.750728 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:22.750450    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:22.750728 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:22.750496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:22.750728 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:22.750577    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:24.747360 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:24.747293    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:24.747875 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:24.747425    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:24.747964 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:24.747940    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:24.748105 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:24.748061    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:24.748356 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:24.748227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:24.748356 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:24.748321    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:26.364757 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:26.364716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:26.365259 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:26.364775    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:26.365259 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:26.364965    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:26.365259 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:26.365026    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret podName:28a43599-1a6c-4f0e-80a9-032b45196140 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:34.365008567 +0000 UTC m=+18.178597408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret") pod "global-pull-secret-syncer-6kw62" (UID: "28a43599-1a6c-4f0e-80a9-032b45196140") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:26.365505 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:26.365486    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:26.365568 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:26.365545    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs podName:5dda191a-363f-44c4-9f13-d1f59108abbc nodeName:}" failed. No retries permitted until 2026-09-16 18:45:34.36552728 +0000 UTC m=+18.179116146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs") pod "network-metrics-daemon-qz4s2" (UID: "5dda191a-363f-44c4-9f13-d1f59108abbc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:26.465501 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:26.465463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmftq\" (UniqueName: \"kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq\") pod \"network-check-target-lfqsq\" (UID: \"9f5c39f3-f783-4d49-b3f1-3cd4392d347c\") " pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:26.465675 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:26.465640    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:26.465675 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:26.465663    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:26.465675 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:26.465676    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rmftq for pod openshift-network-diagnostics/network-check-target-lfqsq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:26.465834 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:26.465744    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq podName:9f5c39f3-f783-4d49-b3f1-3cd4392d347c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:34.465726429 +0000 UTC m=+18.279315285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rmftq" (UniqueName: "kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq") pod "network-check-target-lfqsq" (UID: "9f5c39f3-f783-4d49-b3f1-3cd4392d347c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:26.748909 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:26.748103    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:26.748909 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:26.748249    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:26.748909 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:26.748610    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:26.748909 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:26.748719    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:26.748909 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:26.748763    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:26.748909 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:26.748827    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:28.746741 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:28.746658    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:28.746741 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:28.746676    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:28.746741 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:28.746676    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:28.747320 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:28.746779    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:28.747320 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:28.747205    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:28.747320 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:28.747288    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:30.746640 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:30.746608    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:30.746640 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:30.746634    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:30.747046 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:30.746716    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:30.747046 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:30.746754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:30.747046 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:30.746877    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:30.747046 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:30.746966    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:32.746136 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:32.746088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:32.746575 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:32.746094    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:32.746575 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:32.746222    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:32.746575 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:32.746304    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:32.746575 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:32.746100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:32.746575 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:32.746468    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:34.425698 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:34.425671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:34.426118 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:34.425710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:34.426118 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:34.425814    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:34.426118 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:34.425874    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret podName:28a43599-1a6c-4f0e-80a9-032b45196140 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.425856759 +0000 UTC m=+34.239445614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret") pod "global-pull-secret-syncer-6kw62" (UID: "28a43599-1a6c-4f0e-80a9-032b45196140") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:34.426118 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:34.425814    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:34.426118 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:34.425950    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs podName:5dda191a-363f-44c4-9f13-d1f59108abbc nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.425935387 +0000 UTC m=+34.239524227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs") pod "network-metrics-daemon-qz4s2" (UID: "5dda191a-363f-44c4-9f13-d1f59108abbc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:34.526362 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:34.526331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmftq\" (UniqueName: \"kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq\") pod \"network-check-target-lfqsq\" (UID: \"9f5c39f3-f783-4d49-b3f1-3cd4392d347c\") " pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:34.526488 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:34.526473    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:34.526534 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:34.526494    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:34.526534 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:34.526506    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rmftq for pod openshift-network-diagnostics/network-check-target-lfqsq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:34.526598 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:34.526560    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq podName:9f5c39f3-f783-4d49-b3f1-3cd4392d347c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.526544188 +0000 UTC m=+34.340133030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rmftq" (UniqueName: "kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq") pod "network-check-target-lfqsq" (UID: "9f5c39f3-f783-4d49-b3f1-3cd4392d347c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:34.747069 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:34.746995    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:34.747069 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:34.747005    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:34.747287 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:34.746999    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:34.747287 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:34.747133    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:34.747287 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:34.747194    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:34.747421 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:34.747281    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:36.749521 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:36.748436    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:36.749521 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:36.748578    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:36.749521 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:36.749022    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:36.749521 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:36.749190    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:36.749521 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:36.749247    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:36.749521 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:36.749376    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:38.746724 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.746482    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:38.747572 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.746511    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:38.747572 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:38.746763    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:38.747572 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.746547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:38.747572 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:38.746833    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:38.747572 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:38.746953    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:38.936798 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.936591    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-phj72" event={"ID":"9ae17fba-9a68-4849-a1d4-3c930a0daa61","Type":"ContainerStarted","Data":"ab31a6bc0d88ea40636681182b2c8227e1eec98677f1329a4e8332b604329f92"}
Sep 16 18:45:38.937949 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.937926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb" event={"ID":"3c8f259b-d7cb-4236-a384-474291299c70","Type":"ContainerStarted","Data":"c1b5282568fd4feb7cd9220045444ddec86ce481525170b830854109cb4d7d1e"}
Sep 16 18:45:38.939135 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.939116    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9n9vk" event={"ID":"8aa789e9-6a45-4199-8137-895a502baed7","Type":"ContainerStarted","Data":"98b59af9c9befef1cd92716579ab2c3f57d5b648c101d819e1f9006b8be59785"}
Sep 16 18:45:38.940343 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.940321    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j58qd" event={"ID":"6d00f6df-66e0-472a-abfc-9fd5e349ef69","Type":"ContainerStarted","Data":"e901addeac8f44e667f79a04681390fa0d02688987eecc02b943d6bfe1ca1279"}
Sep 16 18:45:38.941569 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.941550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5qg59" event={"ID":"1bf127e9-9ab0-4581-a467-62bdaa8052f5","Type":"ContainerStarted","Data":"f9b53e2acb4f6e46725803fada58b869e9b3d6ce77730ab656769006ec601f49"}
Sep 16 18:45:38.942956 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.942931    2565 generic.go:356] "Generic (PLEG): container finished" podID="023ad12c-f4a4-43cf-b91a-7004d2eff4ed" containerID="ff225d5e927f98e89b27fa3b2699ccb058981f15a89c0692241033bacc65e475" exitCode=0
Sep 16 18:45:38.943052 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.943006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kq9p" event={"ID":"023ad12c-f4a4-43cf-b91a-7004d2eff4ed","Type":"ContainerDied","Data":"ff225d5e927f98e89b27fa3b2699ccb058981f15a89c0692241033bacc65e475"}
Sep 16 18:45:38.946239 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.946218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5" event={"ID":"686c60bb-f175-4c94-9b73-129291c971b4","Type":"ContainerStarted","Data":"5dbc9bd457e2e1a829ecc9cb15cfec990a1131b50deb61565df69dc96309bcef"}
Sep 16 18:45:38.946328 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.946286    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5" event={"ID":"686c60bb-f175-4c94-9b73-129291c971b4","Type":"ContainerStarted","Data":"daf18a2fb4d7575311e0168738777cc6ecdcd476ba0af5b256384feac0999308"}
Sep 16 18:45:38.946386 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.946335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5" event={"ID":"686c60bb-f175-4c94-9b73-129291c971b4","Type":"ContainerStarted","Data":"704a91ce371aab427d8a08a9de8d789d8a115e693b3eb171b81ff59840aae1d0"}
Sep 16 18:45:38.946386 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.946381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5" event={"ID":"686c60bb-f175-4c94-9b73-129291c971b4","Type":"ContainerStarted","Data":"d3925da5c9a942092a83b4e553a367e3a003c5afa26e66b194546aa3912b3e24"}
Sep 16 18:45:38.946487 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.946430    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5" event={"ID":"686c60bb-f175-4c94-9b73-129291c971b4","Type":"ContainerStarted","Data":"fbae5afccf5517e293edcdb0e063a92720e199fe3127752fd7f4ac696b64e01c"}
Sep 16 18:45:38.946487 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.946478    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5" event={"ID":"686c60bb-f175-4c94-9b73-129291c971b4","Type":"ContainerStarted","Data":"3938d6c7b4e3fa52789bfaec4e250b39aa2b25dd67f1cf33b722e433fcb7993e"}
Sep 16 18:45:38.947487 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.947471    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7kqt2" event={"ID":"4ad64d2a-fd0c-45f9-be97-119db12f0292","Type":"ContainerStarted","Data":"da19972ae6086f13face6c4ce9f4bb2d10a24f2aadaf5d8f97f25acb1b7b3d92"}
Sep 16 18:45:38.970278 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.970241    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-44.ec2.internal" podStartSLOduration=20.970231006 podStartE2EDuration="20.970231006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:21.916398703 +0000 UTC m=+5.729987566" watchObservedRunningTime="2026-09-16 18:45:38.970231006 +0000 UTC m=+22.783819863"
Sep 16 18:45:38.988568 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.988537    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9n9vk" podStartSLOduration=4.037552434 podStartE2EDuration="21.988527335s" totalImagesPullingTime="17.950974901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:38.988075594 +0000 UTC m=+22.801664451" watchObservedRunningTime="2026-09-16 18:45:38.988527335 +0000 UTC m=+22.802116192"
Sep 16 18:45:38.988635 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:38.988591    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-phj72" podStartSLOduration=4.060350549 podStartE2EDuration="21.988587076s" totalImagesPullingTime="17.928236527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:38.970380242 +0000 UTC m=+22.783969102" watchObservedRunningTime="2026-09-16 18:45:38.988587076 +0000 UTC m=+22.802175930"
Sep 16 18:45:39.006652 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:39.006585    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7kqt2" podStartSLOduration=4.010073818 podStartE2EDuration="22.00657526s" totalImagesPullingTime="17.996501442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:39.00651611 +0000 UTC m=+22.820104969" watchObservedRunningTime="2026-09-16 18:45:39.00657526 +0000 UTC m=+22.820164108"
Sep 16 18:45:39.066543 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:39.066516    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5qg59" podStartSLOduration=4.123276557 podStartE2EDuration="22.06650475s" totalImagesPullingTime="17.943228193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:39.048994823 +0000 UTC m=+22.862583680" watchObservedRunningTime="2026-09-16 18:45:39.06650475 +0000 UTC m=+22.880093587"
Sep 16 18:45:39.066833 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:39.066811    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j58qd" podStartSLOduration=7.602406546 podStartE2EDuration="22.066804854s" totalImagesPullingTime="14.464398308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:39.066441422 +0000 UTC m=+22.880030279" watchObservedRunningTime="2026-09-16 18:45:39.066804854 +0000 UTC m=+22.880393712"
Sep 16 18:45:39.313281 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:39.313262    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:45:39.646498 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:39.645989    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:45:39.313276074Z","UUID":"8cb5b1be-51b4-4eed-a480-3aa8ce102cf2","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:45:39.648236 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:39.648205    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:45:39.648390 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:39.648257    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:45:39.951489 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:39.951417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j88k9" event={"ID":"66a5cb54-a1df-40ca-a945-a5378898cf93","Type":"ContainerStarted","Data":"23feb156d970a6ae56680c013ff3b206310d64017c48354406d75a3b3748ec0c"}
Sep 16 18:45:39.954544 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:39.954515    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb" event={"ID":"3c8f259b-d7cb-4236-a384-474291299c70","Type":"ContainerStarted","Data":"dec7b69aba2929dfccc8da9c52eb6d9280a9928c3ccf2c9e14fb9835b05b96fb"}
Sep 16 18:45:40.746527 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:40.746499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:40.746995 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:40.746677    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:40.746995 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:40.746512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:40.746995 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:40.746753    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:40.746995 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:40.746501    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:40.746995 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:40.746879    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:40.891325 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:40.891260    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j58qd"
Sep 16 18:45:40.891870 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:40.891848    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j58qd"
Sep 16 18:45:40.920069 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:40.920022    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j88k9" podStartSLOduration=5.991540463 podStartE2EDuration="23.920003957s" totalImagesPullingTime="17.928463494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:40.005740933 +0000 UTC m=+23.819329792" watchObservedRunningTime="2026-09-16 18:45:40.920003957 +0000 UTC m=+24.733592815"
Sep 16 18:45:40.960443 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:40.960419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5" event={"ID":"686c60bb-f175-4c94-9b73-129291c971b4","Type":"ContainerStarted","Data":"e2fa5bb42d756fa4c2fa6934202f3fdcee08a40fab88ceada010825ba500c134"}
Sep 16 18:45:40.962662 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:40.962627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb" event={"ID":"3c8f259b-d7cb-4236-a384-474291299c70","Type":"ContainerStarted","Data":"a903076abf16a0dcb32c91f5045de7bbda1b3f27ec18a32841e4a49d978d0b6e"}
Sep 16 18:45:41.230715 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:41.230646    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j58qd"
Sep 16 18:45:41.231339 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:41.231311    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j58qd"
Sep 16 18:45:41.255814 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:41.255768    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgvpb" podStartSLOduration=4.39780288 podStartE2EDuration="24.255750229s" totalImagesPullingTime="19.857947349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:40.987879324 +0000 UTC m=+24.801468182" watchObservedRunningTime="2026-09-16 18:45:41.255750229 +0000 UTC m=+25.069339088"
Sep 16 18:45:42.746567 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:42.746463    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:42.746900 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:42.746469    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:42.746900 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:42.746637    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:42.746900 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:42.746493    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:42.746900 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:42.746727    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:42.746900 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:42.746793    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:42.970812 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:42.970780    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5" event={"ID":"686c60bb-f175-4c94-9b73-129291c971b4","Type":"ContainerStarted","Data":"0b9c38e6a1894d9e20fa0831d7dc461dbe5ef2babb470168750d90c260e34f71"}
Sep 16 18:45:42.971066 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:42.971045    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:42.971165 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:42.971116    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:42.985303 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:42.985276    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:42.985559 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:42.985544    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:43.009221 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:43.009182    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5" podStartSLOduration=8.012176748 podStartE2EDuration="26.009170716s" totalImagesPullingTime="17.996993968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:43.008609086 +0000 UTC m=+26.822197944" watchObservedRunningTime="2026-09-16 18:45:43.009170716 +0000 UTC m=+26.822759574"
Sep 16 18:45:43.980059 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:43.980016    2565 generic.go:356] "Generic (PLEG): container finished" podID="023ad12c-f4a4-43cf-b91a-7004d2eff4ed" containerID="8bedc9b713967b5247843e8055f16a8765659318e041fc6be3312866ff04c0e7" exitCode=0
Sep 16 18:45:43.981020 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:43.980075    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kq9p" event={"ID":"023ad12c-f4a4-43cf-b91a-7004d2eff4ed","Type":"ContainerDied","Data":"8bedc9b713967b5247843e8055f16a8765659318e041fc6be3312866ff04c0e7"}
Sep 16 18:45:43.981020 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:43.980255    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:45:44.747043 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:44.746839    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:44.747190 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:44.746839    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:44.747190 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:44.747151    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:44.747190 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:44.747185    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:44.747349 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:44.746875    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:44.747349 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:44.747265    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:44.871525 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:44.871494    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6kw62"]
Sep 16 18:45:44.874312 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:44.874294    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qz4s2"]
Sep 16 18:45:44.887193 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:44.887160    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lfqsq"]
Sep 16 18:45:44.984232 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:44.984168    2565 generic.go:356] "Generic (PLEG): container finished" podID="023ad12c-f4a4-43cf-b91a-7004d2eff4ed" containerID="c663598366671755d5fe4d827253d132211d0ba65b29181e524fda2ea4eaaa74" exitCode=0
Sep 16 18:45:44.984538 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:44.984231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kq9p" event={"ID":"023ad12c-f4a4-43cf-b91a-7004d2eff4ed","Type":"ContainerDied","Data":"c663598366671755d5fe4d827253d132211d0ba65b29181e524fda2ea4eaaa74"}
Sep 16 18:45:44.984538 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:44.984241    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:44.984538 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:44.984335    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:44.984538 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:44.984397    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:44.984538 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:44.984448    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:44.984784 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:44.984553    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:44.984784 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:44.984563    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:45:44.984784 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:44.984591    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:45.989226 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:45.989193    2565 generic.go:356] "Generic (PLEG): container finished" podID="023ad12c-f4a4-43cf-b91a-7004d2eff4ed" containerID="fd9d5d4a754574d1a28b841ad8ea1be94f58d6f847cfc11daf206835da4cd5df" exitCode=0
Sep 16 18:45:45.989520 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:45.989245    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kq9p" event={"ID":"023ad12c-f4a4-43cf-b91a-7004d2eff4ed","Type":"ContainerDied","Data":"fd9d5d4a754574d1a28b841ad8ea1be94f58d6f847cfc11daf206835da4cd5df"}
Sep 16 18:45:46.747535 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:46.747501    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:46.747740 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:46.747632    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:46.747803 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:46.747758    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:46.748099 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:46.747917    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:46.748099 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:46.748028    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:46.748099 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:46.748064    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:47.766177 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:47.766147    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:45:47.766543 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:47.766367    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:45:47.777073 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:47.777024    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5" podUID="686c60bb-f175-4c94-9b73-129291c971b4" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 16 18:45:47.786217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:47.786188    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5" podUID="686c60bb-f175-4c94-9b73-129291c971b4" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 16 18:45:48.746125 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:48.746080    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:48.746368 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:48.746080    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:48.746368 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:48.746222    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:48.746368 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:48.746305    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:48.746368 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:48.746082    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:48.746558 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:48.746436    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:50.448993 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:50.448779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:50.449321 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:50.448989    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:50.449321 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:50.448909    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:50.449321 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:50.449082    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:50.449321 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:50.449088    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs podName:5dda191a-363f-44c4-9f13-d1f59108abbc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:22.44906914 +0000 UTC m=+66.262657980 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs") pod "network-metrics-daemon-qz4s2" (UID: "5dda191a-363f-44c4-9f13-d1f59108abbc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:50.449321 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:50.449166    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret podName:28a43599-1a6c-4f0e-80a9-032b45196140 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:22.449150308 +0000 UTC m=+66.262739145 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret") pod "global-pull-secret-syncer-6kw62" (UID: "28a43599-1a6c-4f0e-80a9-032b45196140") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:50.550151 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:50.550132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmftq\" (UniqueName: \"kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq\") pod \"network-check-target-lfqsq\" (UID: \"9f5c39f3-f783-4d49-b3f1-3cd4392d347c\") " pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:50.550289 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:50.550274    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:50.550337 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:50.550293    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:50.550337 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:50.550303    2565 projected.go:196] Error preparing data for projected volume kube-api-access-rmftq for pod openshift-network-diagnostics/network-check-target-lfqsq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:50.550409 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:50.550341    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq podName:9f5c39f3-f783-4d49-b3f1-3cd4392d347c nodeName:}" failed. No retries permitted until 2026-09-16 18:46:22.550331302 +0000 UTC m=+66.363920138 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rmftq" (UniqueName: "kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq") pod "network-check-target-lfqsq" (UID: "9f5c39f3-f783-4d49-b3f1-3cd4392d347c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:50.746953 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:50.746898    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:50.746953 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:50.746911    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:50.746953 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:50.746916    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:50.747200 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:50.747028    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qz4s2" podUID="5dda191a-363f-44c4-9f13-d1f59108abbc"
Sep 16 18:45:50.747200 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:50.747123    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfqsq" podUID="9f5c39f3-f783-4d49-b3f1-3cd4392d347c"
Sep 16 18:45:50.747200 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:50.747177    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kw62" podUID="28a43599-1a6c-4f0e-80a9-032b45196140"
Sep 16 18:45:51.003753 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.003699    2565 generic.go:356] "Generic (PLEG): container finished" podID="023ad12c-f4a4-43cf-b91a-7004d2eff4ed" containerID="07b7a618224e74a65fc47d6f19804ea4e52dcfa625b207956abdc0afa2c928d0" exitCode=0
Sep 16 18:45:51.003753 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.003748    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kq9p" event={"ID":"023ad12c-f4a4-43cf-b91a-7004d2eff4ed","Type":"ContainerDied","Data":"07b7a618224e74a65fc47d6f19804ea4e52dcfa625b207956abdc0afa2c928d0"}
Sep 16 18:45:51.540908 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.540883    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-44.ec2.internal" event="NodeReady"
Sep 16 18:45:51.541198 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.540978    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:45:51.602621 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.602593    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cc5545569-8jwzj"]
Sep 16 18:45:51.631238 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.631220    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"]
Sep 16 18:45:51.631350 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.631333    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.634509 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.634488    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:45:51.635315 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.635298    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:45:51.635404 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.635328    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:45:51.636669 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.636651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fdc5l\""
Sep 16 18:45:51.651251 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.651234    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5"]
Sep 16 18:45:51.651403 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.651386    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:45:51.660031 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.660014    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:45:51.665446 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.665429    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:45:51.665737 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.665722    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:45:51.665770 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.665739    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:45:51.665988 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.665977    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:45:51.677321 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.677302    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"]
Sep 16 18:45:51.677428 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.677412    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5"
Sep 16 18:45:51.687446 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.687425    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:45:51.692960 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.692942    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cc5545569-8jwzj"]
Sep 16 18:45:51.693049 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.692975    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fxx45"]
Sep 16 18:45:51.693100 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.693049    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.695905 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.695891    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 18:45:51.695983 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.695930    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 18:45:51.696607 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.696594    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 18:45:51.697029 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.697014    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wbgrt\""
Sep 16 18:45:51.698213 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.698201    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 18:45:51.711053 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.711035    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2w45q"]
Sep 16 18:45:51.711232 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.711217    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:51.714993 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.714973    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:45:51.715052 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.715017    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:45:51.715234 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.715222    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fcv7l\""
Sep 16 18:45:51.733159 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.733141    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"]
Sep 16 18:45:51.733259 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.733169    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5"]
Sep 16 18:45:51.733259 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.733181    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2w45q"]
Sep 16 18:45:51.733259 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.733193    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fxx45"]
Sep 16 18:45:51.733259 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.733209    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"]
Sep 16 18:45:51.733259 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.733233    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2w45q"
Sep 16 18:45:51.737537 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.737520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:45:51.737627 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.737598    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wm755\""
Sep 16 18:45:51.737688 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.737674    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:45:51.737741 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.737725    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:45:51.757542 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.757520    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c36ac7b-8947-46c3-8766-072502e904a4-image-registry-private-configuration\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.757609 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.757546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmj6z\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-kube-api-access-zmj6z\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.757609 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.757576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.757609 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.757601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-bound-sa-token\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.757721 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.757636    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0c184af-f384-416e-ac00-d7bd814e84e8-tmp\") pod \"klusterlet-addon-workmgr-87d5f67c4-qphb7\" (UID: \"f0c184af-f384-416e-ac00-d7bd814e84e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:45:51.757721 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.757688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c36ac7b-8947-46c3-8766-072502e904a4-installation-pull-secrets\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.757785 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.757741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c36ac7b-8947-46c3-8766-072502e904a4-ca-trust-extracted\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.757785 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.757769    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c36ac7b-8947-46c3-8766-072502e904a4-registry-certificates\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.757860 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.757799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c36ac7b-8947-46c3-8766-072502e904a4-trusted-ca\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.757860 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.757823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f0c184af-f384-416e-ac00-d7bd814e84e8-klusterlet-config\") pod \"klusterlet-addon-workmgr-87d5f67c4-qphb7\" (UID: \"f0c184af-f384-416e-ac00-d7bd814e84e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:45:51.757860 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.757845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7q24\" (UniqueName: \"kubernetes.io/projected/f0c184af-f384-416e-ac00-d7bd814e84e8-kube-api-access-b7q24\") pod \"klusterlet-addon-workmgr-87d5f67c4-qphb7\" (UID: \"f0c184af-f384-416e-ac00-d7bd814e84e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:45:51.859024 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c36ac7b-8947-46c3-8766-072502e904a4-trusted-ca\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.859129 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859028    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f0c184af-f384-416e-ac00-d7bd814e84e8-klusterlet-config\") pod \"klusterlet-addon-workmgr-87d5f67c4-qphb7\" (UID: \"f0c184af-f384-416e-ac00-d7bd814e84e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:45:51.859129 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859044    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7q24\" (UniqueName: \"kubernetes.io/projected/f0c184af-f384-416e-ac00-d7bd814e84e8-kube-api-access-b7q24\") pod \"klusterlet-addon-workmgr-87d5f67c4-qphb7\" (UID: \"f0c184af-f384-416e-ac00-d7bd814e84e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:45:51.859129 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859070    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:51.859129 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/220b1df3-d68f-44be-8bb9-ffcbcf174803-cert\") pod \"ingress-canary-2w45q\" (UID: \"220b1df3-d68f-44be-8bb9-ffcbcf174803\") " pod="openshift-ingress-canary/ingress-canary-2w45q"
Sep 16 18:45:51.859279 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4076a875-a120-4487-8e18-3c4da6204e62-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.859279 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859199    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/012c41a4-c723-4213-b01f-8ec148e5c96c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5\" (UID: \"012c41a4-c723-4213-b01f-8ec148e5c96c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5"
Sep 16 18:45:51.859279 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r24x\" (UniqueName: \"kubernetes.io/projected/220b1df3-d68f-44be-8bb9-ffcbcf174803-kube-api-access-2r24x\") pod \"ingress-canary-2w45q\" (UID: \"220b1df3-d68f-44be-8bb9-ffcbcf174803\") " pod="openshift-ingress-canary/ingress-canary-2w45q"
Sep 16 18:45:51.859279 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4076a875-a120-4487-8e18-3c4da6204e62-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.859426 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc765\" (UniqueName: \"kubernetes.io/projected/917d490d-2d03-4c46-90a4-f31d97a5bc21-kube-api-access-lc765\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:51.859426 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qch95\" (UniqueName: \"kubernetes.io/projected/4076a875-a120-4487-8e18-3c4da6204e62-kube-api-access-qch95\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.859426 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c36ac7b-8947-46c3-8766-072502e904a4-image-registry-private-configuration\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.859569 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmj6z\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-kube-api-access-zmj6z\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.859569 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.859569 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859515    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-bound-sa-token\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.859569 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0c184af-f384-416e-ac00-d7bd814e84e8-tmp\") pod \"klusterlet-addon-workmgr-87d5f67c4-qphb7\" (UID: \"f0c184af-f384-416e-ac00-d7bd814e84e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:45:51.859569 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917d490d-2d03-4c46-90a4-f31d97a5bc21-tmp-dir\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:51.859829 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c36ac7b-8947-46c3-8766-072502e904a4-installation-pull-secrets\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.859829 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c36ac7b-8947-46c3-8766-072502e904a4-ca-trust-extracted\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.859829 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859645    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4076a875-a120-4487-8e18-3c4da6204e62-hub\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.859829 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/917d490d-2d03-4c46-90a4-f31d97a5bc21-config-volume\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:51.859829 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gwlz\" (UniqueName: \"kubernetes.io/projected/012c41a4-c723-4213-b01f-8ec148e5c96c-kube-api-access-2gwlz\") pod \"managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5\" (UID: \"012c41a4-c723-4213-b01f-8ec148e5c96c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5"
Sep 16 18:45:51.859829 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4076a875-a120-4487-8e18-3c4da6204e62-additional-service-ca\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.859829 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859754    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c36ac7b-8947-46c3-8766-072502e904a4-registry-certificates\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.859829 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4076a875-a120-4487-8e18-3c4da6204e62-ca\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.860207 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.859934    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c36ac7b-8947-46c3-8766-072502e904a4-ca-trust-extracted\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.860207 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.860032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0c184af-f384-416e-ac00-d7bd814e84e8-tmp\") pod \"klusterlet-addon-workmgr-87d5f67c4-qphb7\" (UID: \"f0c184af-f384-416e-ac00-d7bd814e84e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:45:51.860207 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.860051    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c36ac7b-8947-46c3-8766-072502e904a4-trusted-ca\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.860207 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:51.860078    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:51.860207 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:51.860094    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5545569-8jwzj: secret "image-registry-tls" not found
Sep 16 18:45:51.860207 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:51.860164    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls podName:2c36ac7b-8947-46c3-8766-072502e904a4 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:52.360148925 +0000 UTC m=+36.173737763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls") pod "image-registry-cc5545569-8jwzj" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4") : secret "image-registry-tls" not found
Sep 16 18:45:51.860459 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.860343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c36ac7b-8947-46c3-8766-072502e904a4-registry-certificates\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.863052 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.863031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c36ac7b-8947-46c3-8766-072502e904a4-image-registry-private-configuration\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.863152 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.863126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c36ac7b-8947-46c3-8766-072502e904a4-installation-pull-secrets\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.863152 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.863136    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f0c184af-f384-416e-ac00-d7bd814e84e8-klusterlet-config\") pod \"klusterlet-addon-workmgr-87d5f67c4-qphb7\" (UID: \"f0c184af-f384-416e-ac00-d7bd814e84e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:45:51.875804 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.875780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7q24\" (UniqueName: \"kubernetes.io/projected/f0c184af-f384-416e-ac00-d7bd814e84e8-kube-api-access-b7q24\") pod \"klusterlet-addon-workmgr-87d5f67c4-qphb7\" (UID: \"f0c184af-f384-416e-ac00-d7bd814e84e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:45:51.875890 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.875848    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmj6z\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-kube-api-access-zmj6z\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.876987 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.876972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-bound-sa-token\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:51.959119 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.959062    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:45:51.960251 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gwlz\" (UniqueName: \"kubernetes.io/projected/012c41a4-c723-4213-b01f-8ec148e5c96c-kube-api-access-2gwlz\") pod \"managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5\" (UID: \"012c41a4-c723-4213-b01f-8ec148e5c96c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5"
Sep 16 18:45:51.960251 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4076a875-a120-4487-8e18-3c4da6204e62-additional-service-ca\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.960368 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960273    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4076a875-a120-4487-8e18-3c4da6204e62-ca\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.960415 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:51.960466 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960414    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/220b1df3-d68f-44be-8bb9-ffcbcf174803-cert\") pod \"ingress-canary-2w45q\" (UID: \"220b1df3-d68f-44be-8bb9-ffcbcf174803\") " pod="openshift-ingress-canary/ingress-canary-2w45q"
Sep 16 18:45:51.960466 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4076a875-a120-4487-8e18-3c4da6204e62-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.960466 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:51.960453    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:51.960606 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/012c41a4-c723-4213-b01f-8ec148e5c96c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5\" (UID: \"012c41a4-c723-4213-b01f-8ec148e5c96c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5"
Sep 16 18:45:51.960606 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:51.960501    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls podName:917d490d-2d03-4c46-90a4-f31d97a5bc21 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:52.460485575 +0000 UTC m=+36.274074425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls") pod "dns-default-fxx45" (UID: "917d490d-2d03-4c46-90a4-f31d97a5bc21") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:51.960606 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r24x\" (UniqueName: \"kubernetes.io/projected/220b1df3-d68f-44be-8bb9-ffcbcf174803-kube-api-access-2r24x\") pod \"ingress-canary-2w45q\" (UID: \"220b1df3-d68f-44be-8bb9-ffcbcf174803\") " pod="openshift-ingress-canary/ingress-canary-2w45q"
Sep 16 18:45:51.960606 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4076a875-a120-4487-8e18-3c4da6204e62-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.960606 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc765\" (UniqueName: \"kubernetes.io/projected/917d490d-2d03-4c46-90a4-f31d97a5bc21-kube-api-access-lc765\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:51.960856 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qch95\" (UniqueName: \"kubernetes.io/projected/4076a875-a120-4487-8e18-3c4da6204e62-kube-api-access-qch95\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.960856 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917d490d-2d03-4c46-90a4-f31d97a5bc21-tmp-dir\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:51.960951 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4076a875-a120-4487-8e18-3c4da6204e62-additional-service-ca\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.960997 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.960963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917d490d-2d03-4c46-90a4-f31d97a5bc21-tmp-dir\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:51.961323 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.961167    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4076a875-a120-4487-8e18-3c4da6204e62-hub\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.961323 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.961201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/917d490d-2d03-4c46-90a4-f31d97a5bc21-config-volume\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:51.961735 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.961713    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/917d490d-2d03-4c46-90a4-f31d97a5bc21-config-volume\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:51.962696 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.962666    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4076a875-a120-4487-8e18-3c4da6204e62-ca\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.962827 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.962810    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4076a875-a120-4487-8e18-3c4da6204e62-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.963326 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.963303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/012c41a4-c723-4213-b01f-8ec148e5c96c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5\" (UID: \"012c41a4-c723-4213-b01f-8ec148e5c96c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5"
Sep 16 18:45:51.963444 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.963427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4076a875-a120-4487-8e18-3c4da6204e62-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.963623 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.963601    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/220b1df3-d68f-44be-8bb9-ffcbcf174803-cert\") pod \"ingress-canary-2w45q\" (UID: \"220b1df3-d68f-44be-8bb9-ffcbcf174803\") " pod="openshift-ingress-canary/ingress-canary-2w45q"
Sep 16 18:45:51.963728 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.963710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4076a875-a120-4487-8e18-3c4da6204e62-hub\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.974771 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.974744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qch95\" (UniqueName: \"kubernetes.io/projected/4076a875-a120-4487-8e18-3c4da6204e62-kube-api-access-qch95\") pod \"cluster-proxy-proxy-agent-656f9cd754-nr9sd\" (UID: \"4076a875-a120-4487-8e18-3c4da6204e62\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:51.974993 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.974969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc765\" (UniqueName: \"kubernetes.io/projected/917d490d-2d03-4c46-90a4-f31d97a5bc21-kube-api-access-lc765\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:51.977238 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.977215    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r24x\" (UniqueName: \"kubernetes.io/projected/220b1df3-d68f-44be-8bb9-ffcbcf174803-kube-api-access-2r24x\") pod \"ingress-canary-2w45q\" (UID: \"220b1df3-d68f-44be-8bb9-ffcbcf174803\") " pod="openshift-ingress-canary/ingress-canary-2w45q"
Sep 16 18:45:51.977566 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.977551    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gwlz\" (UniqueName: \"kubernetes.io/projected/012c41a4-c723-4213-b01f-8ec148e5c96c-kube-api-access-2gwlz\") pod \"managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5\" (UID: \"012c41a4-c723-4213-b01f-8ec148e5c96c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5"
Sep 16 18:45:51.995823 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:51.995802    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5"
Sep 16 18:45:52.001956 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.001939    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:45:52.009073 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.009053    2565 generic.go:356] "Generic (PLEG): container finished" podID="023ad12c-f4a4-43cf-b91a-7004d2eff4ed" containerID="e3b1598885b5e742d8a19e0e86afe6845e6a2e33653cf367aba26578907cfd9b" exitCode=0
Sep 16 18:45:52.009165 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.009129    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kq9p" event={"ID":"023ad12c-f4a4-43cf-b91a-7004d2eff4ed","Type":"ContainerDied","Data":"e3b1598885b5e742d8a19e0e86afe6845e6a2e33653cf367aba26578907cfd9b"}
Sep 16 18:45:52.040947 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.040929    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2w45q"
Sep 16 18:45:52.286260 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.286010    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"]
Sep 16 18:45:52.294066 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:52.294034    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c184af_f384_416e_ac00_d7bd814e84e8.slice/crio-18f0b9940b1da77f4dd6926af3f05ebe35ce4b524756768c794f51028cd35ad7 WatchSource:0}: Error finding container 18f0b9940b1da77f4dd6926af3f05ebe35ce4b524756768c794f51028cd35ad7: Status 404 returned error can't find the container with id 18f0b9940b1da77f4dd6926af3f05ebe35ce4b524756768c794f51028cd35ad7
Sep 16 18:45:52.312808 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.312786    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5"]
Sep 16 18:45:52.315043 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:52.315013    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod012c41a4_c723_4213_b01f_8ec148e5c96c.slice/crio-24a56da4be994ce125311d218eb4a2dd5096db9b7e809edc06d86e3e57ad5f60 WatchSource:0}: Error finding container 24a56da4be994ce125311d218eb4a2dd5096db9b7e809edc06d86e3e57ad5f60: Status 404 returned error can't find the container with id 24a56da4be994ce125311d218eb4a2dd5096db9b7e809edc06d86e3e57ad5f60
Sep 16 18:45:52.358718 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.358701    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"]
Sep 16 18:45:52.360686 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:52.360660    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4076a875_a120_4487_8e18_3c4da6204e62.slice/crio-1c14fd8c6afa64acbf56fcf558b030d8b99bc11843b293a25b7ef80a57e41733 WatchSource:0}: Error finding container 1c14fd8c6afa64acbf56fcf558b030d8b99bc11843b293a25b7ef80a57e41733: Status 404 returned error can't find the container with id 1c14fd8c6afa64acbf56fcf558b030d8b99bc11843b293a25b7ef80a57e41733
Sep 16 18:45:52.363715 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.363697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:52.363865 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:52.363840    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:52.363865 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:52.363855    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5545569-8jwzj: secret "image-registry-tls" not found
Sep 16 18:45:52.363923 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:52.363904    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls podName:2c36ac7b-8947-46c3-8766-072502e904a4 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:53.363883857 +0000 UTC m=+37.177472706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls") pod "image-registry-cc5545569-8jwzj" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4") : secret "image-registry-tls" not found
Sep 16 18:45:52.378091 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.378070    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2w45q"]
Sep 16 18:45:52.380283 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:45:52.380263    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220b1df3_d68f_44be_8bb9_ffcbcf174803.slice/crio-1e0342339d4e392eb4859b44dfd3f943743a341ff3024b8a132032c3e23894e0 WatchSource:0}: Error finding container 1e0342339d4e392eb4859b44dfd3f943743a341ff3024b8a132032c3e23894e0: Status 404 returned error can't find the container with id 1e0342339d4e392eb4859b44dfd3f943743a341ff3024b8a132032c3e23894e0
Sep 16 18:45:52.464226 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.464202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:52.464343 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:52.464330    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:52.464421 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:52.464376    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls podName:917d490d-2d03-4c46-90a4-f31d97a5bc21 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:53.464363671 +0000 UTC m=+37.277952507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls") pod "dns-default-fxx45" (UID: "917d490d-2d03-4c46-90a4-f31d97a5bc21") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:52.749241 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.749189    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:45:52.749908 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.749281    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:45:52.749908 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.749382    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:45:52.753037 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.753020    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:52.753155 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.753020    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hcvfn\""
Sep 16 18:45:52.753155 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.753054    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:52.753359 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.753344    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:52.753390 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.753344    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:52.754534 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:52.754520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kzr79\""
Sep 16 18:45:53.013048 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:53.012973    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2w45q" event={"ID":"220b1df3-d68f-44be-8bb9-ffcbcf174803","Type":"ContainerStarted","Data":"1e0342339d4e392eb4859b44dfd3f943743a341ff3024b8a132032c3e23894e0"}
Sep 16 18:45:53.016995 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:53.016964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kq9p" event={"ID":"023ad12c-f4a4-43cf-b91a-7004d2eff4ed","Type":"ContainerStarted","Data":"f147d82285ba36209b0f582eb2ec702b1ff1c95f9b4e1eec40a75ecb03a3303a"}
Sep 16 18:45:53.018200 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:53.018179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd" event={"ID":"4076a875-a120-4487-8e18-3c4da6204e62","Type":"ContainerStarted","Data":"1c14fd8c6afa64acbf56fcf558b030d8b99bc11843b293a25b7ef80a57e41733"}
Sep 16 18:45:53.019322 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:53.019293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5" event={"ID":"012c41a4-c723-4213-b01f-8ec148e5c96c","Type":"ContainerStarted","Data":"24a56da4be994ce125311d218eb4a2dd5096db9b7e809edc06d86e3e57ad5f60"}
Sep 16 18:45:53.020498 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:53.020469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7" event={"ID":"f0c184af-f384-416e-ac00-d7bd814e84e8","Type":"ContainerStarted","Data":"18f0b9940b1da77f4dd6926af3f05ebe35ce4b524756768c794f51028cd35ad7"}
Sep 16 18:45:53.045576 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:53.045505    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4kq9p" podStartSLOduration=8.584246334 podStartE2EDuration="36.045489429s" totalImagesPullingTime="27.461243095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:53.044324931 +0000 UTC m=+36.857913789" watchObservedRunningTime="2026-09-16 18:45:53.045489429 +0000 UTC m=+36.859078288"
Sep 16 18:45:53.373163 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:53.372445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:53.373163 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:53.372663    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:53.373163 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:53.372681    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5545569-8jwzj: secret "image-registry-tls" not found
Sep 16 18:45:53.373163 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:53.372742    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls podName:2c36ac7b-8947-46c3-8766-072502e904a4 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:55.372722154 +0000 UTC m=+39.186311012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls") pod "image-registry-cc5545569-8jwzj" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4") : secret "image-registry-tls" not found
Sep 16 18:45:53.473703 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:53.473088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:53.473703 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:53.473316    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:53.473703 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:53.473382    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls podName:917d490d-2d03-4c46-90a4-f31d97a5bc21 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:55.473365438 +0000 UTC m=+39.286954300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls") pod "dns-default-fxx45" (UID: "917d490d-2d03-4c46-90a4-f31d97a5bc21") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:55.389055 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:55.389018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:55.389466 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:55.389216    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:55.389466 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:55.389238    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5545569-8jwzj: secret "image-registry-tls" not found
Sep 16 18:45:55.389466 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:55.389313    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls podName:2c36ac7b-8947-46c3-8766-072502e904a4 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:59.38928981 +0000 UTC m=+43.202878669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls") pod "image-registry-cc5545569-8jwzj" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4") : secret "image-registry-tls" not found
Sep 16 18:45:55.490307 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:55.490241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:55.490445 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:55.490404    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:55.490505 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:55.490474    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls podName:917d490d-2d03-4c46-90a4-f31d97a5bc21 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:59.490453864 +0000 UTC m=+43.304042722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls") pod "dns-default-fxx45" (UID: "917d490d-2d03-4c46-90a4-f31d97a5bc21") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:59.417576 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:59.417533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:45:59.417965 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:59.417708    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:59.417965 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:59.417729    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5545569-8jwzj: secret "image-registry-tls" not found
Sep 16 18:45:59.417965 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:59.417793    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls podName:2c36ac7b-8947-46c3-8766-072502e904a4 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.417777679 +0000 UTC m=+51.231366515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls") pod "image-registry-cc5545569-8jwzj" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4") : secret "image-registry-tls" not found
Sep 16 18:45:59.517973 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:45:59.517937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:45:59.518159 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:59.518099    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:59.518283 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:45:59.518177    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls podName:917d490d-2d03-4c46-90a4-f31d97a5bc21 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.518156558 +0000 UTC m=+51.331745403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls") pod "dns-default-fxx45" (UID: "917d490d-2d03-4c46-90a4-f31d97a5bc21") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:01.047031 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:01.046999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5" event={"ID":"012c41a4-c723-4213-b01f-8ec148e5c96c","Type":"ContainerStarted","Data":"072a8f50bb93baf96273375a64c45a52474d6a3b761f89e57e71582bcb6335fd"}
Sep 16 18:46:01.048546 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:01.048518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7" event={"ID":"f0c184af-f384-416e-ac00-d7bd814e84e8","Type":"ContainerStarted","Data":"29d6b2e4716ce49c0f227e3e56d9b4a493a601dd8af6647e4923f368303a4e44"}
Sep 16 18:46:01.048716 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:01.048700    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:46:01.050331 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:01.050308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2w45q" event={"ID":"220b1df3-d68f-44be-8bb9-ffcbcf174803","Type":"ContainerStarted","Data":"8b2c4be353659f1c1c144525e212318079403da91cd724ec2711e4a64e34d893"}
Sep 16 18:46:01.050669 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:01.050649    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7"
Sep 16 18:46:01.052437 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:01.052419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd" event={"ID":"4076a875-a120-4487-8e18-3c4da6204e62","Type":"ContainerStarted","Data":"f4396b1a9cef24f914fb74bc3af94c73052a440f5f4a9245c5732c78c434222d"}
Sep 16 18:46:01.052535 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:01.052446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd" event={"ID":"4076a875-a120-4487-8e18-3c4da6204e62","Type":"ContainerStarted","Data":"6eb9db16c92564acf0b0223d524926afa3159c7bcd8076c15cc26f68f0bf78e5"}
Sep 16 18:46:01.080280 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:01.080240    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6647cd6d4d-tzcr5" podStartSLOduration=27.721068413 podStartE2EDuration="36.08022877s" totalImagesPullingTime="8.359160357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:01.079862532 +0000 UTC m=+44.893451389" watchObservedRunningTime="2026-09-16 18:46:01.08022877 +0000 UTC m=+44.893817619"
Sep 16 18:46:01.106865 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:01.106828    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2w45q" podStartSLOduration=1.8184150159999999 podStartE2EDuration="10.106815065s" totalImagesPullingTime="8.288400049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:01.104793327 +0000 UTC m=+44.918382184" watchObservedRunningTime="2026-09-16 18:46:01.106815065 +0000 UTC m=+44.920403914"
Sep 16 18:46:01.126679 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:01.126636    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-87d5f67c4-qphb7" podStartSLOduration=27.752769311 podStartE2EDuration="36.126620665s" totalImagesPullingTime="8.373851354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:01.125807998 +0000 UTC m=+44.939396857" watchObservedRunningTime="2026-09-16 18:46:01.126620665 +0000 UTC m=+44.940209523"
Sep 16 18:46:02.061540 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:02.061498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd" event={"ID":"4076a875-a120-4487-8e18-3c4da6204e62","Type":"ContainerStarted","Data":"2d62797651fc64cdcbdb8d07e033da9acf9cfb61150366be7e0ddf405d098c29"}
Sep 16 18:46:02.085928 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:02.085881    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd" podStartSLOduration=28.778695282 podStartE2EDuration="37.08586851s" totalImagesPullingTime="8.307173228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:02.08444382 +0000 UTC m=+45.898032678" watchObservedRunningTime="2026-09-16 18:46:02.08586851 +0000 UTC m=+45.899457362"
Sep 16 18:46:07.474746 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:07.474706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:46:07.475205 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:07.474816    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:07.475205 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:07.474829    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5545569-8jwzj: secret "image-registry-tls" not found
Sep 16 18:46:07.475205 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:07.474888    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls podName:2c36ac7b-8947-46c3-8766-072502e904a4 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:23.474874425 +0000 UTC m=+67.288463261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls") pod "image-registry-cc5545569-8jwzj" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4") : secret "image-registry-tls" not found
Sep 16 18:46:07.575170 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:07.575141    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:46:07.575305 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:07.575276    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:07.575363 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:07.575350    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls podName:917d490d-2d03-4c46-90a4-f31d97a5bc21 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:23.57533478 +0000 UTC m=+67.388923616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls") pod "dns-default-fxx45" (UID: "917d490d-2d03-4c46-90a4-f31d97a5bc21") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:17.787082 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:17.787053    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-thbb5"
Sep 16 18:46:22.475661 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:22.475622    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:46:22.475661 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:22.475665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:46:22.478932 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:22.478912    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:22.478989 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:22.478942    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:22.486139 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:22.486099    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:46:22.486229 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:22.486192    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs podName:5dda191a-363f-44c4-9f13-d1f59108abbc nodeName:}" failed. No retries permitted until 2026-09-16 18:47:26.486172715 +0000 UTC m=+130.299761551 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs") pod "network-metrics-daemon-qz4s2" (UID: "5dda191a-363f-44c4-9f13-d1f59108abbc") : secret "metrics-daemon-secret" not found
Sep 16 18:46:22.490068 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:22.490049    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28a43599-1a6c-4f0e-80a9-032b45196140-original-pull-secret\") pod \"global-pull-secret-syncer-6kw62\" (UID: \"28a43599-1a6c-4f0e-80a9-032b45196140\") " pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:46:22.576800 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:22.576770    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmftq\" (UniqueName: \"kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq\") pod \"network-check-target-lfqsq\" (UID: \"9f5c39f3-f783-4d49-b3f1-3cd4392d347c\") " pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:46:22.580048 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:22.580028    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:22.589811 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:22.589788    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:22.600449 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:22.600416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmftq\" (UniqueName: \"kubernetes.io/projected/9f5c39f3-f783-4d49-b3f1-3cd4392d347c-kube-api-access-rmftq\") pod \"network-check-target-lfqsq\" (UID: \"9f5c39f3-f783-4d49-b3f1-3cd4392d347c\") " pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:46:22.762273 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:22.762218    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hcvfn\""
Sep 16 18:46:22.764332 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:22.764314    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kw62"
Sep 16 18:46:22.769785 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:22.769765    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:46:23.038373 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:23.038307    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6kw62"]
Sep 16 18:46:23.040966 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:46:23.040942    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a43599_1a6c_4f0e_80a9_032b45196140.slice/crio-c5ba47a02f6674929f2d121ac844ad5e76ec3073be89c12579a5a6bec78f44e1 WatchSource:0}: Error finding container c5ba47a02f6674929f2d121ac844ad5e76ec3073be89c12579a5a6bec78f44e1: Status 404 returned error can't find the container with id c5ba47a02f6674929f2d121ac844ad5e76ec3073be89c12579a5a6bec78f44e1
Sep 16 18:46:23.055742 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:23.055712    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lfqsq"]
Sep 16 18:46:23.057903 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:46:23.057882    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5c39f3_f783_4d49_b3f1_3cd4392d347c.slice/crio-b56458cb20544d4829f5bd3f9872172b7ed2533f5aaa352499bd6a2d1bade9e6 WatchSource:0}: Error finding container b56458cb20544d4829f5bd3f9872172b7ed2533f5aaa352499bd6a2d1bade9e6: Status 404 returned error can't find the container with id b56458cb20544d4829f5bd3f9872172b7ed2533f5aaa352499bd6a2d1bade9e6
Sep 16 18:46:23.126500 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:23.126471    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6kw62" event={"ID":"28a43599-1a6c-4f0e-80a9-032b45196140","Type":"ContainerStarted","Data":"c5ba47a02f6674929f2d121ac844ad5e76ec3073be89c12579a5a6bec78f44e1"}
Sep 16 18:46:23.127582 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:23.127561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lfqsq" event={"ID":"9f5c39f3-f783-4d49-b3f1-3cd4392d347c","Type":"ContainerStarted","Data":"b56458cb20544d4829f5bd3f9872172b7ed2533f5aaa352499bd6a2d1bade9e6"}
Sep 16 18:46:23.483671 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:23.483639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:46:23.483974 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:23.483790    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:23.483974 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:23.483804    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5545569-8jwzj: secret "image-registry-tls" not found
Sep 16 18:46:23.483974 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:23.483857    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls podName:2c36ac7b-8947-46c3-8766-072502e904a4 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:55.48384161 +0000 UTC m=+99.297430447 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls") pod "image-registry-cc5545569-8jwzj" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4") : secret "image-registry-tls" not found
Sep 16 18:46:23.584494 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:23.584464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:46:23.584649 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:23.584631    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:23.584721 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:23.584699    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls podName:917d490d-2d03-4c46-90a4-f31d97a5bc21 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:55.584679162 +0000 UTC m=+99.398268017 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls") pod "dns-default-fxx45" (UID: "917d490d-2d03-4c46-90a4-f31d97a5bc21") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:28.146726 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:28.146692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6kw62" event={"ID":"28a43599-1a6c-4f0e-80a9-032b45196140","Type":"ContainerStarted","Data":"7598ac30380bac7dfb942ab17c3983cdcea3db2ffdc58c974976c477935f113c"}
Sep 16 18:46:28.148311 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:28.148284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lfqsq" event={"ID":"9f5c39f3-f783-4d49-b3f1-3cd4392d347c","Type":"ContainerStarted","Data":"d64e186513da9b972496d2b4f37c8962f3eabee2f735af080439c8f45941c824"}
Sep 16 18:46:28.148444 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:28.148413    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:46:28.168430 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:28.168381    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6kw62" podStartSLOduration=66.937297072 podStartE2EDuration="1m11.16836481s" totalImagesPullingTime="4.231067738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:28.166840906 +0000 UTC m=+71.980429763" watchObservedRunningTime="2026-09-16 18:46:28.16836481 +0000 UTC m=+71.981953671"
Sep 16 18:46:28.195851 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:28.195797    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lfqsq" podStartSLOduration=66.985178284 podStartE2EDuration="1m11.195784366s" totalImagesPullingTime="4.210606082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:28.195551008 +0000 UTC m=+72.009139859" watchObservedRunningTime="2026-09-16 18:46:28.195784366 +0000 UTC m=+72.009373224"
Sep 16 18:46:55.498761 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:55.498723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls\") pod \"image-registry-cc5545569-8jwzj\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") " pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:46:55.499219 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:55.498851    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:55.499219 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:55.498865    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5545569-8jwzj: secret "image-registry-tls" not found
Sep 16 18:46:55.499219 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:55.498924    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls podName:2c36ac7b-8947-46c3-8766-072502e904a4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:59.498906086 +0000 UTC m=+163.312494941 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls") pod "image-registry-cc5545569-8jwzj" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4") : secret "image-registry-tls" not found
Sep 16 18:46:55.599078 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:55.599051    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:46:55.599249 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:55.599224    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:55.599333 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:46:55.599320    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls podName:917d490d-2d03-4c46-90a4-f31d97a5bc21 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:59.599298751 +0000 UTC m=+163.412887591 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls") pod "dns-default-fxx45" (UID: "917d490d-2d03-4c46-90a4-f31d97a5bc21") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:59.153785 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:46:59.153749    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lfqsq"
Sep 16 18:47:07.294621 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.294577    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rjwhv"]
Sep 16 18:47:07.298191 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.298173    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.305736 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.305710    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:47:07.305878 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.305797    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:47:07.306259 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.306233    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:47:07.306389 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.306236    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:47:07.306389 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.306236    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h44qh\""
Sep 16 18:47:07.332116 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.332087    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rjwhv"]
Sep 16 18:47:07.374703 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.374680    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29016bee-c5d1-4e37-a13d-bce6b454ae8d-crio-socket\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.374837 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.374732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvbv\" (UniqueName: \"kubernetes.io/projected/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-api-access-vlvbv\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.374837 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.374801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29016bee-c5d1-4e37-a13d-bce6b454ae8d-data-volume\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.374934 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.374847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.374934 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.374880    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.476115 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.476084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29016bee-c5d1-4e37-a13d-bce6b454ae8d-data-volume\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.476236 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.476139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.476236 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.476159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.476236 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:07.476226    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:07.476368 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:07.476254    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:07.476368 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:07.476286    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm podName:29016bee-c5d1-4e37-a13d-bce6b454ae8d nodeName:}" failed. No retries permitted until 2026-09-16 18:47:07.97627124 +0000 UTC m=+111.789860080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjwhv" (UID: "29016bee-c5d1-4e37-a13d-bce6b454ae8d") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:07.476368 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:07.476322    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls podName:29016bee-c5d1-4e37-a13d-bce6b454ae8d nodeName:}" failed. No retries permitted until 2026-09-16 18:47:07.976306717 +0000 UTC m=+111.789895557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjwhv" (UID: "29016bee-c5d1-4e37-a13d-bce6b454ae8d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:07.476368 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.476354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29016bee-c5d1-4e37-a13d-bce6b454ae8d-crio-socket\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.476577 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.476427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29016bee-c5d1-4e37-a13d-bce6b454ae8d-data-volume\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.476577 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.476432    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29016bee-c5d1-4e37-a13d-bce6b454ae8d-crio-socket\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.476577 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.476461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvbv\" (UniqueName: \"kubernetes.io/projected/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-api-access-vlvbv\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.491001 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.490970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvbv\" (UniqueName: \"kubernetes.io/projected/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-api-access-vlvbv\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.979778 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.979749    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.979884 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:07.979785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:07.979884 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:07.979860    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:07.979958 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:07.979888    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:07.979958 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:07.979930    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm podName:29016bee-c5d1-4e37-a13d-bce6b454ae8d nodeName:}" failed. No retries permitted until 2026-09-16 18:47:08.979912734 +0000 UTC m=+112.793501649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjwhv" (UID: "29016bee-c5d1-4e37-a13d-bce6b454ae8d") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:07.979958 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:07.979945    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls podName:29016bee-c5d1-4e37-a13d-bce6b454ae8d nodeName:}" failed. No retries permitted until 2026-09-16 18:47:08.979938027 +0000 UTC m=+112.793526890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjwhv" (UID: "29016bee-c5d1-4e37-a13d-bce6b454ae8d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:08.986657 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:08.986623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:08.986657 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:08.986659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:08.987075 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:08.986755    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:08.987075 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:08.986823    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls podName:29016bee-c5d1-4e37-a13d-bce6b454ae8d nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.986805814 +0000 UTC m=+114.800394650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjwhv" (UID: "29016bee-c5d1-4e37-a13d-bce6b454ae8d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:08.987075 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:08.986756    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:08.987075 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:08.986860    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm podName:29016bee-c5d1-4e37-a13d-bce6b454ae8d nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.986851716 +0000 UTC m=+114.800440552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjwhv" (UID: "29016bee-c5d1-4e37-a13d-bce6b454ae8d") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:09.013818 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:09.013796    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9n9vk_8aa789e9-6a45-4199-8137-895a502baed7/dns-node-resolver/0.log"
Sep 16 18:47:10.406215 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:10.406187    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-phj72_9ae17fba-9a68-4849-a1d4-3c930a0daa61/node-ca/0.log"
Sep 16 18:47:11.001060 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:11.001021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:11.001060 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:11.001067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:11.001253 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:11.001177    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:11.001253 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:11.001196    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:11.001253 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:11.001253    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls podName:29016bee-c5d1-4e37-a13d-bce6b454ae8d nodeName:}" failed. No retries permitted until 2026-09-16 18:47:15.001237669 +0000 UTC m=+118.814826509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjwhv" (UID: "29016bee-c5d1-4e37-a13d-bce6b454ae8d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:11.001359 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:11.001268    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm podName:29016bee-c5d1-4e37-a13d-bce6b454ae8d nodeName:}" failed. No retries permitted until 2026-09-16 18:47:15.001261163 +0000 UTC m=+118.814849999 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjwhv" (UID: "29016bee-c5d1-4e37-a13d-bce6b454ae8d") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:11.005997 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:11.005974    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2w45q_220b1df3-d68f-44be-8bb9-ffcbcf174803/serve-healthcheck-canary/0.log"
Sep 16 18:47:15.028805 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:15.028766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:15.028805 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:15.028810    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:15.029380 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:15.028900    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:15.029380 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:15.028917    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:15.029380 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:15.028963    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm podName:29016bee-c5d1-4e37-a13d-bce6b454ae8d nodeName:}" failed. No retries permitted until 2026-09-16 18:47:23.028943162 +0000 UTC m=+126.842531998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjwhv" (UID: "29016bee-c5d1-4e37-a13d-bce6b454ae8d") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:15.029380 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:15.028991    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls podName:29016bee-c5d1-4e37-a13d-bce6b454ae8d nodeName:}" failed. No retries permitted until 2026-09-16 18:47:23.028975839 +0000 UTC m=+126.842564679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjwhv" (UID: "29016bee-c5d1-4e37-a13d-bce6b454ae8d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:23.093140 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:23.093078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:23.093140 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:23.093142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:23.093890 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:23.093643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29016bee-c5d1-4e37-a13d-bce6b454ae8d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:23.095685 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:23.095652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29016bee-c5d1-4e37-a13d-bce6b454ae8d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjwhv\" (UID: \"29016bee-c5d1-4e37-a13d-bce6b454ae8d\") " pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:23.210897 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:23.210864    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h44qh\""
Sep 16 18:47:23.218425 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:23.218402    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rjwhv"
Sep 16 18:47:23.500615 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:23.500591    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rjwhv"]
Sep 16 18:47:23.503811 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:47:23.503787    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29016bee_c5d1_4e37_a13d_bce6b454ae8d.slice/crio-e595f9376bae5dd028ea18e64298058c8aca84a28034761eac478d3d15f1abbc WatchSource:0}: Error finding container e595f9376bae5dd028ea18e64298058c8aca84a28034761eac478d3d15f1abbc: Status 404 returned error can't find the container with id e595f9376bae5dd028ea18e64298058c8aca84a28034761eac478d3d15f1abbc
Sep 16 18:47:24.312004 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:24.311973    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rjwhv" event={"ID":"29016bee-c5d1-4e37-a13d-bce6b454ae8d","Type":"ContainerStarted","Data":"2f87d3145c77f5de31fa25a6a01f8b09b7ceb34e4c77ffdd218f8fbdfe95fd3d"}
Sep 16 18:47:24.312301 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:24.312028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rjwhv" event={"ID":"29016bee-c5d1-4e37-a13d-bce6b454ae8d","Type":"ContainerStarted","Data":"63a8778eef36146e46c145af64d34275fc58649557d4bd7cf5dbc4179f82fb31"}
Sep 16 18:47:24.312301 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:24.312050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rjwhv" event={"ID":"29016bee-c5d1-4e37-a13d-bce6b454ae8d","Type":"ContainerStarted","Data":"e595f9376bae5dd028ea18e64298058c8aca84a28034761eac478d3d15f1abbc"}
Sep 16 18:47:26.320166 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:26.320132    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rjwhv" event={"ID":"29016bee-c5d1-4e37-a13d-bce6b454ae8d","Type":"ContainerStarted","Data":"4d087ffbf3b590718e61ea4d6ddddd9ad66fd09d677647f192eacfa9db6ddedf"}
Sep 16 18:47:26.342677 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:26.342629    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rjwhv" podStartSLOduration=17.289627519 podStartE2EDuration="19.342610964s" totalImagesPullingTime="2.052983445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:26.341316645 +0000 UTC m=+130.154905503" watchObservedRunningTime="2026-09-16 18:47:26.342610964 +0000 UTC m=+130.156199821"
Sep 16 18:47:26.520139 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:26.520092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:47:26.522359 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:26.522333    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dda191a-363f-44c4-9f13-d1f59108abbc-metrics-certs\") pod \"network-metrics-daemon-qz4s2\" (UID: \"5dda191a-363f-44c4-9f13-d1f59108abbc\") " pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:47:26.671508 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:26.671447    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kzr79\""
Sep 16 18:47:26.678772 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:26.678751    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qz4s2"
Sep 16 18:47:26.955255 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:26.955151    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qz4s2"]
Sep 16 18:47:26.957555 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:47:26.957530    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dda191a_363f_44c4_9f13_d1f59108abbc.slice/crio-692e9240b69aea2e29a4dd3a559037acac05bd4e78f59b0e63acc79880c12214 WatchSource:0}: Error finding container 692e9240b69aea2e29a4dd3a559037acac05bd4e78f59b0e63acc79880c12214: Status 404 returned error can't find the container with id 692e9240b69aea2e29a4dd3a559037acac05bd4e78f59b0e63acc79880c12214
Sep 16 18:47:27.325158 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:27.325062    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qz4s2" event={"ID":"5dda191a-363f-44c4-9f13-d1f59108abbc","Type":"ContainerStarted","Data":"692e9240b69aea2e29a4dd3a559037acac05bd4e78f59b0e63acc79880c12214"}
Sep 16 18:47:28.329573 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:28.329546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qz4s2" event={"ID":"5dda191a-363f-44c4-9f13-d1f59108abbc","Type":"ContainerStarted","Data":"72cca38c3745e52065bd4b8f7b23ae71fed05acbc1ba85a567c1a0e173319745"}
Sep 16 18:47:28.329847 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:28.329597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qz4s2" event={"ID":"5dda191a-363f-44c4-9f13-d1f59108abbc","Type":"ContainerStarted","Data":"332f7de3980ab99d773ff7a4afa265c71510229dbc275f004599943d72abe8a5"}
Sep 16 18:47:32.003300 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:32.003260    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd" podUID="4076a875-a120-4487-8e18-3c4da6204e62" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:47:42.003009 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:42.002963    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd" podUID="4076a875-a120-4487-8e18-3c4da6204e62" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:47:47.889260 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:47.889196    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qz4s2" podStartSLOduration=149.703801254 podStartE2EDuration="2m30.889178457s" totalImagesPullingTime="1.185377203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:28.356051025 +0000 UTC m=+132.169639879" watchObservedRunningTime="2026-09-16 18:47:47.889178457 +0000 UTC m=+151.702767314"
Sep 16 18:47:47.890254 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:47.890232    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v4mwx"]
Sep 16 18:47:47.894786 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:47.894767    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:47.898012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:47.897988    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:47:47.898160 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:47.898036    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:47:47.899157 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:47.899135    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:47:47.904817 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:47.904793    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:47:47.904997 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:47.904870    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:47:47.906428 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:47.906408    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:47:47.909644 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:47.909625    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7p4vc\""
Sep 16 18:47:48.066137 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.066093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-tls\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.066261 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.066148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-sys\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.066261 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.066173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhhcz\" (UniqueName: \"kubernetes.io/projected/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-kube-api-access-jhhcz\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.066261 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.066193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.066261 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.066218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-metrics-client-ca\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.066261 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.066244    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-root\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.066261 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.066260    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-wtmp\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.066468 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.066348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-accelerators-collector-config\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.066468 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.066402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-textfile\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167065 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.166989    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-tls\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167065 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-sys\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167258 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhhcz\" (UniqueName: \"kubernetes.io/projected/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-kube-api-access-jhhcz\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167258 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167258 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-sys\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167258 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-metrics-client-ca\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167258 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:48.167223    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:47:48.167453 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167267    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-root\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167453 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167229    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-root\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167453 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:48.167299    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-tls podName:aa4b2ef6-bca2-4442-b89c-d4236b49c2db nodeName:}" failed. No retries permitted until 2026-09-16 18:47:48.667282338 +0000 UTC m=+152.480871174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-tls") pod "node-exporter-v4mwx" (UID: "aa4b2ef6-bca2-4442-b89c-d4236b49c2db") : secret "node-exporter-tls" not found
Sep 16 18:47:48.167453 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-wtmp\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167630 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-accelerators-collector-config\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167630 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-textfile\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167630 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-wtmp\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167808 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167772    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-metrics-client-ca\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.167946 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-textfile\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.168012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.167984    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-accelerators-collector-config\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.169725 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.169701    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.194408 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.194381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhhcz\" (UniqueName: \"kubernetes.io/projected/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-kube-api-access-jhhcz\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.672088 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.672048    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-tls\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.674662 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.674626    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa4b2ef6-bca2-4442-b89c-d4236b49c2db-node-exporter-tls\") pod \"node-exporter-v4mwx\" (UID: \"aa4b2ef6-bca2-4442-b89c-d4236b49c2db\") " pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.804513 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:48.804486    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v4mwx"
Sep 16 18:47:48.816759 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:47:48.816729    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4b2ef6_bca2_4442_b89c_d4236b49c2db.slice/crio-6792c5bd05e9ec5755194ce45775b0fbc8ef09a0613aeb0c1bf643c4421d6074 WatchSource:0}: Error finding container 6792c5bd05e9ec5755194ce45775b0fbc8ef09a0613aeb0c1bf643c4421d6074: Status 404 returned error can't find the container with id 6792c5bd05e9ec5755194ce45775b0fbc8ef09a0613aeb0c1bf643c4421d6074
Sep 16 18:47:49.395152 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:49.395120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v4mwx" event={"ID":"aa4b2ef6-bca2-4442-b89c-d4236b49c2db","Type":"ContainerStarted","Data":"6792c5bd05e9ec5755194ce45775b0fbc8ef09a0613aeb0c1bf643c4421d6074"}
Sep 16 18:47:50.399596 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:50.399562    2565 generic.go:356] "Generic (PLEG): container finished" podID="aa4b2ef6-bca2-4442-b89c-d4236b49c2db" containerID="f58b8685e27d6a84d223918df0caffa80da1c378df1a5b4eb7b407027a9efcd8" exitCode=0
Sep 16 18:47:50.399906 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:50.399620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v4mwx" event={"ID":"aa4b2ef6-bca2-4442-b89c-d4236b49c2db","Type":"ContainerDied","Data":"f58b8685e27d6a84d223918df0caffa80da1c378df1a5b4eb7b407027a9efcd8"}
Sep 16 18:47:51.405589 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:51.405547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v4mwx" event={"ID":"aa4b2ef6-bca2-4442-b89c-d4236b49c2db","Type":"ContainerStarted","Data":"c61c49aad52c636d57462110220ff9b200e28309a286a52ff9f1e086beb61ca1"}
Sep 16 18:47:51.406006 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:51.405605    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v4mwx" event={"ID":"aa4b2ef6-bca2-4442-b89c-d4236b49c2db","Type":"ContainerStarted","Data":"e02e2299b02ca723f099b920a6d7669050492c6bf03e0e61981010d7a56ae06b"}
Sep 16 18:47:51.472492 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:51.472442    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v4mwx" podStartSLOduration=3.554886577 podStartE2EDuration="4.472427198s" totalImagesPullingTime="859.605369ms" totalInitContainerRuntime="57.935252ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:51.472048506 +0000 UTC m=+155.285637363" watchObservedRunningTime="2026-09-16 18:47:51.472427198 +0000 UTC m=+155.286016056"
Sep 16 18:47:52.003685 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:52.003650    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd" podUID="4076a875-a120-4487-8e18-3c4da6204e62" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:47:52.003863 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:52.003723    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd"
Sep 16 18:47:52.004199 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:52.004163    2565 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"2d62797651fc64cdcbdb8d07e033da9acf9cfb61150366be7e0ddf405d098c29"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 16 18:47:52.004310 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:52.004220    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd" podUID="4076a875-a120-4487-8e18-3c4da6204e62" containerName="service-proxy" containerID="cri-o://2d62797651fc64cdcbdb8d07e033da9acf9cfb61150366be7e0ddf405d098c29" gracePeriod=30
Sep 16 18:47:52.411344 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:52.411267    2565 generic.go:356] "Generic (PLEG): container finished" podID="4076a875-a120-4487-8e18-3c4da6204e62" containerID="2d62797651fc64cdcbdb8d07e033da9acf9cfb61150366be7e0ddf405d098c29" exitCode=2
Sep 16 18:47:52.411666 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:52.411333    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd" event={"ID":"4076a875-a120-4487-8e18-3c4da6204e62","Type":"ContainerDied","Data":"2d62797651fc64cdcbdb8d07e033da9acf9cfb61150366be7e0ddf405d098c29"}
Sep 16 18:47:52.411666 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:52.411395    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656f9cd754-nr9sd" event={"ID":"4076a875-a120-4487-8e18-3c4da6204e62","Type":"ContainerStarted","Data":"63640ffe89b07740ac5f6b7d5902ce28b212aab05f7e374318af3694186f97fa"}
Sep 16 18:47:54.640388 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:54.640340    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-cc5545569-8jwzj" podUID="2c36ac7b-8947-46c3-8766-072502e904a4"
Sep 16 18:47:54.734563 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:54.734526    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fxx45" podUID="917d490d-2d03-4c46-90a4-f31d97a5bc21"
Sep 16 18:47:55.423000 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:55.422973    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:47:55.423147 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:55.422973    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fxx45"
Sep 16 18:47:56.223715 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.223679    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cc5545569-8jwzj"]
Sep 16 18:47:56.224065 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:47:56.223857    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-cc5545569-8jwzj" podUID="2c36ac7b-8947-46c3-8766-072502e904a4"
Sep 16 18:47:56.426533 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.426500    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:47:56.433869 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.433845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:47:56.525616 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.525559    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c36ac7b-8947-46c3-8766-072502e904a4-trusted-ca\") pod \"2c36ac7b-8947-46c3-8766-072502e904a4\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") "
Sep 16 18:47:56.525616 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.525596    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-bound-sa-token\") pod \"2c36ac7b-8947-46c3-8766-072502e904a4\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") "
Sep 16 18:47:56.525756 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.525623    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c36ac7b-8947-46c3-8766-072502e904a4-installation-pull-secrets\") pod \"2c36ac7b-8947-46c3-8766-072502e904a4\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") "
Sep 16 18:47:56.525756 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.525747    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmj6z\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-kube-api-access-zmj6z\") pod \"2c36ac7b-8947-46c3-8766-072502e904a4\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") "
Sep 16 18:47:56.525845 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.525785    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c36ac7b-8947-46c3-8766-072502e904a4-image-registry-private-configuration\") pod \"2c36ac7b-8947-46c3-8766-072502e904a4\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") "
Sep 16 18:47:56.525845 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.525806    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c36ac7b-8947-46c3-8766-072502e904a4-ca-trust-extracted\") pod \"2c36ac7b-8947-46c3-8766-072502e904a4\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") "
Sep 16 18:47:56.525845 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.525830    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c36ac7b-8947-46c3-8766-072502e904a4-registry-certificates\") pod \"2c36ac7b-8947-46c3-8766-072502e904a4\" (UID: \"2c36ac7b-8947-46c3-8766-072502e904a4\") "
Sep 16 18:47:56.526037 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.526003    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c36ac7b-8947-46c3-8766-072502e904a4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2c36ac7b-8947-46c3-8766-072502e904a4" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:56.526218 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.526191    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c36ac7b-8947-46c3-8766-072502e904a4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2c36ac7b-8947-46c3-8766-072502e904a4" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:56.526341 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.526302    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c36ac7b-8947-46c3-8766-072502e904a4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2c36ac7b-8947-46c3-8766-072502e904a4" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:56.528136 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.528091    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c36ac7b-8947-46c3-8766-072502e904a4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2c36ac7b-8947-46c3-8766-072502e904a4" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:56.528330 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.528302    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2c36ac7b-8947-46c3-8766-072502e904a4" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:56.528394 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.528361    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-kube-api-access-zmj6z" (OuterVolumeSpecName: "kube-api-access-zmj6z") pod "2c36ac7b-8947-46c3-8766-072502e904a4" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4"). InnerVolumeSpecName "kube-api-access-zmj6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:56.528446 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.528385    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c36ac7b-8947-46c3-8766-072502e904a4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2c36ac7b-8947-46c3-8766-072502e904a4" (UID: "2c36ac7b-8947-46c3-8766-072502e904a4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:56.627068 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.627037    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-bound-sa-token\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:47:56.627068 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.627067    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c36ac7b-8947-46c3-8766-072502e904a4-installation-pull-secrets\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:47:56.627208 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.627076    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmj6z\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-kube-api-access-zmj6z\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:47:56.627208 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.627086    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c36ac7b-8947-46c3-8766-072502e904a4-image-registry-private-configuration\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:47:56.627208 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.627095    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c36ac7b-8947-46c3-8766-072502e904a4-ca-trust-extracted\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:47:56.627208 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.627126    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c36ac7b-8947-46c3-8766-072502e904a4-registry-certificates\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:47:56.627208 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:56.627140    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c36ac7b-8947-46c3-8766-072502e904a4-trusted-ca\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:47:57.429412 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:57.429378    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc5545569-8jwzj"
Sep 16 18:47:57.483698 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:57.483669    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cc5545569-8jwzj"]
Sep 16 18:47:57.491630 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:57.491607    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cc5545569-8jwzj"]
Sep 16 18:47:57.533557 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:57.533533    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c36ac7b-8947-46c3-8766-072502e904a4-registry-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:47:58.751308 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:58.751278    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c36ac7b-8947-46c3-8766-072502e904a4" path="/var/lib/kubelet/pods/2c36ac7b-8947-46c3-8766-072502e904a4/volumes"
Sep 16 18:47:59.648146 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:59.648095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:47:59.650615 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:59.650592    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917d490d-2d03-4c46-90a4-f31d97a5bc21-metrics-tls\") pod \"dns-default-fxx45\" (UID: \"917d490d-2d03-4c46-90a4-f31d97a5bc21\") " pod="openshift-dns/dns-default-fxx45"
Sep 16 18:47:59.927201 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:59.927142    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fcv7l\""
Sep 16 18:47:59.934642 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:47:59.934619    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fxx45"
Sep 16 18:48:00.232541 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:48:00.232515    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fxx45"]
Sep 16 18:48:00.233760 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:48:00.233733    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917d490d_2d03_4c46_90a4_f31d97a5bc21.slice/crio-74bfb79de3781272b441af742b010f3f6d258003138f2c9aa7c89a84507ee6db WatchSource:0}: Error finding container 74bfb79de3781272b441af742b010f3f6d258003138f2c9aa7c89a84507ee6db: Status 404 returned error can't find the container with id 74bfb79de3781272b441af742b010f3f6d258003138f2c9aa7c89a84507ee6db
Sep 16 18:48:00.444336 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:48:00.444295    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fxx45" event={"ID":"917d490d-2d03-4c46-90a4-f31d97a5bc21","Type":"ContainerStarted","Data":"74bfb79de3781272b441af742b010f3f6d258003138f2c9aa7c89a84507ee6db"}
Sep 16 18:48:02.453330 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:48:02.453291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fxx45" event={"ID":"917d490d-2d03-4c46-90a4-f31d97a5bc21","Type":"ContainerStarted","Data":"d5c235e55d9e96bb75b5ade545eacd2cd44848fba8f81125e3fac34532fd842f"}
Sep 16 18:48:02.453781 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:48:02.453364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fxx45" event={"ID":"917d490d-2d03-4c46-90a4-f31d97a5bc21","Type":"ContainerStarted","Data":"a9b9d0a8147a6333a1a652a3cb776339fc27a6732a3e97ea25e472aebc669beb"}
Sep 16 18:48:02.453781 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:48:02.453444    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fxx45"
Sep 16 18:48:02.482632 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:48:02.482587    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fxx45" podStartSLOduration=129.82552261 podStartE2EDuration="2m11.482572856s" totalImagesPullingTime="1.657050246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:02.481966732 +0000 UTC m=+166.295555589" watchObservedRunningTime="2026-09-16 18:48:02.482572856 +0000 UTC m=+166.296161788"
Sep 16 18:48:12.458971 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:48:12.458881    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fxx45"
Sep 16 18:50:16.602387 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:16.602350    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:50:58.014494 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.014457    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-hn5dl"]
Sep 16 18:50:58.017665 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.017647    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:50:58.020215 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.020191    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:50:58.020527 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.020510    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:50:58.021407 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.021389    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vw4pb\""
Sep 16 18:50:58.021407 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.021402    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 18:50:58.021524 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.021449    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:50:58.021524 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.021408    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:50:58.030734 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.030714    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-hn5dl"]
Sep 16 18:50:58.070844 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.070823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvsbl\" (UniqueName: \"kubernetes.io/projected/011898a9-6f4c-4b7c-9f8b-394b5dc55132-kube-api-access-kvsbl\") pod \"keda-operator-579c8958f7-hn5dl\" (UID: \"011898a9-6f4c-4b7c-9f8b-394b5dc55132\") " pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:50:58.070941 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.070850    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/011898a9-6f4c-4b7c-9f8b-394b5dc55132-certificates\") pod \"keda-operator-579c8958f7-hn5dl\" (UID: \"011898a9-6f4c-4b7c-9f8b-394b5dc55132\") " pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:50:58.070941 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.070885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/011898a9-6f4c-4b7c-9f8b-394b5dc55132-cabundle0\") pod \"keda-operator-579c8958f7-hn5dl\" (UID: \"011898a9-6f4c-4b7c-9f8b-394b5dc55132\") " pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:50:58.171133 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.171093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/011898a9-6f4c-4b7c-9f8b-394b5dc55132-certificates\") pod \"keda-operator-579c8958f7-hn5dl\" (UID: \"011898a9-6f4c-4b7c-9f8b-394b5dc55132\") " pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:50:58.171233 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.171171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/011898a9-6f4c-4b7c-9f8b-394b5dc55132-cabundle0\") pod \"keda-operator-579c8958f7-hn5dl\" (UID: \"011898a9-6f4c-4b7c-9f8b-394b5dc55132\") " pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:50:58.171233 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.171206    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvsbl\" (UniqueName: \"kubernetes.io/projected/011898a9-6f4c-4b7c-9f8b-394b5dc55132-kube-api-access-kvsbl\") pod \"keda-operator-579c8958f7-hn5dl\" (UID: \"011898a9-6f4c-4b7c-9f8b-394b5dc55132\") " pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:50:58.171233 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:50:58.171215    2565 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:50:58.171381 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:50:58.171236    2565 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:50:58.171381 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:50:58.171258    2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-hn5dl: references non-existent secret key: ca.crt
Sep 16 18:50:58.171381 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:50:58.171315    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/011898a9-6f4c-4b7c-9f8b-394b5dc55132-certificates podName:011898a9-6f4c-4b7c-9f8b-394b5dc55132 nodeName:}" failed. No retries permitted until 2026-09-16 18:50:58.671301813 +0000 UTC m=+342.484890654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/011898a9-6f4c-4b7c-9f8b-394b5dc55132-certificates") pod "keda-operator-579c8958f7-hn5dl" (UID: "011898a9-6f4c-4b7c-9f8b-394b5dc55132") : references non-existent secret key: ca.crt
Sep 16 18:50:58.171834 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.171810    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/011898a9-6f4c-4b7c-9f8b-394b5dc55132-cabundle0\") pod \"keda-operator-579c8958f7-hn5dl\" (UID: \"011898a9-6f4c-4b7c-9f8b-394b5dc55132\") " pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:50:58.182937 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.182914    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvsbl\" (UniqueName: \"kubernetes.io/projected/011898a9-6f4c-4b7c-9f8b-394b5dc55132-kube-api-access-kvsbl\") pod \"keda-operator-579c8958f7-hn5dl\" (UID: \"011898a9-6f4c-4b7c-9f8b-394b5dc55132\") " pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:50:58.228986 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.228965    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"]
Sep 16 18:50:58.231898 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.231883    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:50:58.234234 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.234212    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 18:50:58.242095 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.242074    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"]
Sep 16 18:50:58.271606 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.271557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gbd2\" (UniqueName: \"kubernetes.io/projected/c2bd93be-b169-42ac-aa5f-60f82d8e97d6-kube-api-access-5gbd2\") pod \"keda-metrics-apiserver-77857b7b98-bqg9p\" (UID: \"c2bd93be-b169-42ac-aa5f-60f82d8e97d6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:50:58.271606 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.271581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c2bd93be-b169-42ac-aa5f-60f82d8e97d6-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-bqg9p\" (UID: \"c2bd93be-b169-42ac-aa5f-60f82d8e97d6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:50:58.271724 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.271610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c2bd93be-b169-42ac-aa5f-60f82d8e97d6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-bqg9p\" (UID: \"c2bd93be-b169-42ac-aa5f-60f82d8e97d6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:50:58.372721 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.372696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gbd2\" (UniqueName: \"kubernetes.io/projected/c2bd93be-b169-42ac-aa5f-60f82d8e97d6-kube-api-access-5gbd2\") pod \"keda-metrics-apiserver-77857b7b98-bqg9p\" (UID: \"c2bd93be-b169-42ac-aa5f-60f82d8e97d6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:50:58.372721 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.372723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c2bd93be-b169-42ac-aa5f-60f82d8e97d6-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-bqg9p\" (UID: \"c2bd93be-b169-42ac-aa5f-60f82d8e97d6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:50:58.372876 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.372752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c2bd93be-b169-42ac-aa5f-60f82d8e97d6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-bqg9p\" (UID: \"c2bd93be-b169-42ac-aa5f-60f82d8e97d6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:50:58.373047 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.373028    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c2bd93be-b169-42ac-aa5f-60f82d8e97d6-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-bqg9p\" (UID: \"c2bd93be-b169-42ac-aa5f-60f82d8e97d6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:50:58.375280 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.375259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c2bd93be-b169-42ac-aa5f-60f82d8e97d6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-bqg9p\" (UID: \"c2bd93be-b169-42ac-aa5f-60f82d8e97d6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:50:58.384317 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.384296    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gbd2\" (UniqueName: \"kubernetes.io/projected/c2bd93be-b169-42ac-aa5f-60f82d8e97d6-kube-api-access-5gbd2\") pod \"keda-metrics-apiserver-77857b7b98-bqg9p\" (UID: \"c2bd93be-b169-42ac-aa5f-60f82d8e97d6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:50:58.541225 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.541169    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:50:58.674712 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.674685    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/011898a9-6f4c-4b7c-9f8b-394b5dc55132-certificates\") pod \"keda-operator-579c8958f7-hn5dl\" (UID: \"011898a9-6f4c-4b7c-9f8b-394b5dc55132\") " pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:50:58.676765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.676745    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/011898a9-6f4c-4b7c-9f8b-394b5dc55132-certificates\") pod \"keda-operator-579c8958f7-hn5dl\" (UID: \"011898a9-6f4c-4b7c-9f8b-394b5dc55132\") " pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:50:58.810231 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.810138    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"]
Sep 16 18:50:58.812612 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:50:58.812586    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bd93be_b169_42ac_aa5f_60f82d8e97d6.slice/crio-c761aa0cbd09028ff16e8250b037cd69c43cc9dff4bde47ff2d33e9d01c98b73 WatchSource:0}: Error finding container c761aa0cbd09028ff16e8250b037cd69c43cc9dff4bde47ff2d33e9d01c98b73: Status 404 returned error can't find the container with id c761aa0cbd09028ff16e8250b037cd69c43cc9dff4bde47ff2d33e9d01c98b73
Sep 16 18:50:58.813793 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.813775    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:50:58.927211 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:58.927188    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:50:59.024489 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:59.024463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p" event={"ID":"c2bd93be-b169-42ac-aa5f-60f82d8e97d6","Type":"ContainerStarted","Data":"c761aa0cbd09028ff16e8250b037cd69c43cc9dff4bde47ff2d33e9d01c98b73"}
Sep 16 18:50:59.198880 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:50:59.198848    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-hn5dl"]
Sep 16 18:50:59.201432 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:50:59.201406    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011898a9_6f4c_4b7c_9f8b_394b5dc55132.slice/crio-43236b14ad568c3e791227c247410cd7835a4e3fef65787270d362e10d09172e WatchSource:0}: Error finding container 43236b14ad568c3e791227c247410cd7835a4e3fef65787270d362e10d09172e: Status 404 returned error can't find the container with id 43236b14ad568c3e791227c247410cd7835a4e3fef65787270d362e10d09172e
Sep 16 18:51:00.031719 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:51:00.031606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-hn5dl" event={"ID":"011898a9-6f4c-4b7c-9f8b-394b5dc55132","Type":"ContainerStarted","Data":"43236b14ad568c3e791227c247410cd7835a4e3fef65787270d362e10d09172e"}
Sep 16 18:51:03.047118 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:51:03.047064    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p" event={"ID":"c2bd93be-b169-42ac-aa5f-60f82d8e97d6","Type":"ContainerStarted","Data":"261a8f5fe5223cb5cbca0a242aa6b35135fc78138672846f867fc9338a84c8d2"}
Sep 16 18:51:03.047499 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:51:03.047271    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:51:03.048657 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:51:03.048631    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-hn5dl" event={"ID":"011898a9-6f4c-4b7c-9f8b-394b5dc55132","Type":"ContainerStarted","Data":"856dd04228821a2992afd5e4857a872b7e03932f0139ce8fc3e21665fe15fbf6"}
Sep 16 18:51:03.048768 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:51:03.048679    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:51:03.146258 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:51:03.146214    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p" podStartSLOduration=1.335601864 podStartE2EDuration="5.146197808s" totalImagesPullingTime="3.810595944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:03.0961954 +0000 UTC m=+346.909784257" watchObservedRunningTime="2026-09-16 18:51:03.146197808 +0000 UTC m=+346.959786669"
Sep 16 18:51:14.055862 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:51:14.055833    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bqg9p"
Sep 16 18:51:14.096200 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:51:14.096162    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-hn5dl" podStartSLOduration=13.674051646 podStartE2EDuration="17.096147856s" totalImagesPullingTime="3.42209621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:03.147623528 +0000 UTC m=+346.961212385" watchObservedRunningTime="2026-09-16 18:51:14.096147856 +0000 UTC m=+357.909736712"
Sep 16 18:51:24.053497 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:51:24.053470    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-hn5dl"
Sep 16 18:52:20.436497 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:20.436464    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"]
Sep 16 18:52:20.439380 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:20.439364    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"
Sep 16 18:52:20.442171 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:20.442148    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:52:20.442988 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:20.442970    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:52:20.443082 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:20.443041    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-rtkxw\""
Sep 16 18:52:20.443082 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:20.443070    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:52:20.451037 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:20.451008    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"]
Sep 16 18:52:20.517061 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:20.517027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c19469e-0474-4f15-9d5d-0b3231fc283d-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-s6lkw\" (UID: \"8c19469e-0474-4f15-9d5d-0b3231fc283d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"
Sep 16 18:52:20.517198 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:20.517128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d64s\" (UniqueName: \"kubernetes.io/projected/8c19469e-0474-4f15-9d5d-0b3231fc283d-kube-api-access-4d64s\") pod \"llmisvc-controller-manager-645f9c8fd6-s6lkw\" (UID: \"8c19469e-0474-4f15-9d5d-0b3231fc283d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"
Sep 16 18:52:20.617479 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:20.617450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d64s\" (UniqueName: \"kubernetes.io/projected/8c19469e-0474-4f15-9d5d-0b3231fc283d-kube-api-access-4d64s\") pod \"llmisvc-controller-manager-645f9c8fd6-s6lkw\" (UID: \"8c19469e-0474-4f15-9d5d-0b3231fc283d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"
Sep 16 18:52:20.617585 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:20.617487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c19469e-0474-4f15-9d5d-0b3231fc283d-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-s6lkw\" (UID: \"8c19469e-0474-4f15-9d5d-0b3231fc283d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"
Sep 16 18:52:20.617623 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:52:20.617587    2565 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 18:52:20.617655 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:52:20.617647    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c19469e-0474-4f15-9d5d-0b3231fc283d-cert podName:8c19469e-0474-4f15-9d5d-0b3231fc283d nodeName:}" failed. No retries permitted until 2026-09-16 18:52:21.11762894 +0000 UTC m=+424.931217778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8c19469e-0474-4f15-9d5d-0b3231fc283d-cert") pod "llmisvc-controller-manager-645f9c8fd6-s6lkw" (UID: "8c19469e-0474-4f15-9d5d-0b3231fc283d") : secret "llmisvc-webhook-server-cert" not found
Sep 16 18:52:20.626162 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:20.626143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d64s\" (UniqueName: \"kubernetes.io/projected/8c19469e-0474-4f15-9d5d-0b3231fc283d-kube-api-access-4d64s\") pod \"llmisvc-controller-manager-645f9c8fd6-s6lkw\" (UID: \"8c19469e-0474-4f15-9d5d-0b3231fc283d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"
Sep 16 18:52:21.120398 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:21.120365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c19469e-0474-4f15-9d5d-0b3231fc283d-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-s6lkw\" (UID: \"8c19469e-0474-4f15-9d5d-0b3231fc283d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"
Sep 16 18:52:21.122611 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:21.122585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c19469e-0474-4f15-9d5d-0b3231fc283d-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-s6lkw\" (UID: \"8c19469e-0474-4f15-9d5d-0b3231fc283d\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"
Sep 16 18:52:21.348455 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:21.348419    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"
Sep 16 18:52:21.615195 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:21.615174    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"]
Sep 16 18:52:21.617574 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:52:21.617542    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8c19469e_0474_4f15_9d5d_0b3231fc283d.slice/crio-36df2f1a44e3c35d003ba4369d580c6a68acbc644824c871daaa48255ab16db2 WatchSource:0}: Error finding container 36df2f1a44e3c35d003ba4369d580c6a68acbc644824c871daaa48255ab16db2: Status 404 returned error can't find the container with id 36df2f1a44e3c35d003ba4369d580c6a68acbc644824c871daaa48255ab16db2
Sep 16 18:52:22.304473 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:22.304436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw" event={"ID":"8c19469e-0474-4f15-9d5d-0b3231fc283d","Type":"ContainerStarted","Data":"36df2f1a44e3c35d003ba4369d580c6a68acbc644824c871daaa48255ab16db2"}
Sep 16 18:52:24.313690 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:24.313660    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw" event={"ID":"8c19469e-0474-4f15-9d5d-0b3231fc283d","Type":"ContainerStarted","Data":"79ec5885a95dc79aa8c5f4a3726cdab67a1419a437c0cde0dfcbaa4a2f307ab0"}
Sep 16 18:52:24.313984 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:24.313779    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"
Sep 16 18:52:24.339734 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:24.339660    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw" podStartSLOduration=1.893031455 podStartE2EDuration="4.339646131s" totalImagesPullingTime="2.446614676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:24.338570606 +0000 UTC m=+428.152159463" watchObservedRunningTime="2026-09-16 18:52:24.339646131 +0000 UTC m=+428.153234987"
Sep 16 18:52:55.318467 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:52:55.318363    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-s6lkw"
Sep 16 18:55:17.788162 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:17.788130    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"]
Sep 16 18:55:17.791052 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:17.791034    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"
Sep 16 18:55:17.794431 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:17.794405    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:55:17.794542 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:17.794466    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mv896\""
Sep 16 18:55:17.803544 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:17.803520    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"]
Sep 16 18:55:17.967449 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:17.967420    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c8f1176d-5e04-4b50-ac50-8444754bcaf3-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-hnmqn\" (UID: \"c8f1176d-5e04-4b50-ac50-8444754bcaf3\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"
Sep 16 18:55:17.967547 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:17.967464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh4n5\" (UniqueName: \"kubernetes.io/projected/c8f1176d-5e04-4b50-ac50-8444754bcaf3-kube-api-access-nh4n5\") pod \"seaweedfs-tls-custom-6644bf5d6c-hnmqn\" (UID: \"c8f1176d-5e04-4b50-ac50-8444754bcaf3\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"
Sep 16 18:55:18.068155 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:18.068067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c8f1176d-5e04-4b50-ac50-8444754bcaf3-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-hnmqn\" (UID: \"c8f1176d-5e04-4b50-ac50-8444754bcaf3\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"
Sep 16 18:55:18.068155 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:18.068133    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh4n5\" (UniqueName: \"kubernetes.io/projected/c8f1176d-5e04-4b50-ac50-8444754bcaf3-kube-api-access-nh4n5\") pod \"seaweedfs-tls-custom-6644bf5d6c-hnmqn\" (UID: \"c8f1176d-5e04-4b50-ac50-8444754bcaf3\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"
Sep 16 18:55:18.068433 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:18.068411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c8f1176d-5e04-4b50-ac50-8444754bcaf3-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-hnmqn\" (UID: \"c8f1176d-5e04-4b50-ac50-8444754bcaf3\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"
Sep 16 18:55:18.079058 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:18.079036    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh4n5\" (UniqueName: \"kubernetes.io/projected/c8f1176d-5e04-4b50-ac50-8444754bcaf3-kube-api-access-nh4n5\") pod \"seaweedfs-tls-custom-6644bf5d6c-hnmqn\" (UID: \"c8f1176d-5e04-4b50-ac50-8444754bcaf3\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"
Sep 16 18:55:18.099972 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:18.099952    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"
Sep 16 18:55:18.377902 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:18.377828    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"]
Sep 16 18:55:18.381974 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:55:18.381942    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f1176d_5e04_4b50_ac50_8444754bcaf3.slice/crio-135a594a84d37bb1fd6c7d1dda623696605204530226af53e5d4bf39c4401ca2 WatchSource:0}: Error finding container 135a594a84d37bb1fd6c7d1dda623696605204530226af53e5d4bf39c4401ca2: Status 404 returned error can't find the container with id 135a594a84d37bb1fd6c7d1dda623696605204530226af53e5d4bf39c4401ca2
Sep 16 18:55:18.896761 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:18.896687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn" event={"ID":"c8f1176d-5e04-4b50-ac50-8444754bcaf3","Type":"ContainerStarted","Data":"135a594a84d37bb1fd6c7d1dda623696605204530226af53e5d4bf39c4401ca2"}
Sep 16 18:55:21.911269 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:21.911220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn" event={"ID":"c8f1176d-5e04-4b50-ac50-8444754bcaf3","Type":"ContainerStarted","Data":"7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8"}
Sep 16 18:55:21.930998 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:21.930942    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn" podStartSLOduration=2.258242386 podStartE2EDuration="4.930924519s" totalImagesPullingTime="2.672682133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:21.930512838 +0000 UTC m=+605.744101696" watchObservedRunningTime="2026-09-16 18:55:21.930924519 +0000 UTC m=+605.744513376"
Sep 16 18:55:25.347751 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:25.347701    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"]
Sep 16 18:55:25.348280 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:25.347958    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn" podUID="c8f1176d-5e04-4b50-ac50-8444754bcaf3" containerName="seaweedfs-tls-custom" containerID="cri-o://7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8" gracePeriod=30
Sep 16 18:55:26.572627 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.572603    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"
Sep 16 18:55:26.725563 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.725542    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh4n5\" (UniqueName: \"kubernetes.io/projected/c8f1176d-5e04-4b50-ac50-8444754bcaf3-kube-api-access-nh4n5\") pod \"c8f1176d-5e04-4b50-ac50-8444754bcaf3\" (UID: \"c8f1176d-5e04-4b50-ac50-8444754bcaf3\") "
Sep 16 18:55:26.725723 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.725589    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c8f1176d-5e04-4b50-ac50-8444754bcaf3-data\") pod \"c8f1176d-5e04-4b50-ac50-8444754bcaf3\" (UID: \"c8f1176d-5e04-4b50-ac50-8444754bcaf3\") "
Sep 16 18:55:26.726886 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.726859    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f1176d-5e04-4b50-ac50-8444754bcaf3-data" (OuterVolumeSpecName: "data") pod "c8f1176d-5e04-4b50-ac50-8444754bcaf3" (UID: "c8f1176d-5e04-4b50-ac50-8444754bcaf3"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:26.727678 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.727653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f1176d-5e04-4b50-ac50-8444754bcaf3-kube-api-access-nh4n5" (OuterVolumeSpecName: "kube-api-access-nh4n5") pod "c8f1176d-5e04-4b50-ac50-8444754bcaf3" (UID: "c8f1176d-5e04-4b50-ac50-8444754bcaf3"). InnerVolumeSpecName "kube-api-access-nh4n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:26.826897 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.826870    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh4n5\" (UniqueName: \"kubernetes.io/projected/c8f1176d-5e04-4b50-ac50-8444754bcaf3-kube-api-access-nh4n5\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:26.826897 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.826893    2565 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c8f1176d-5e04-4b50-ac50-8444754bcaf3-data\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:55:26.930345 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.930312    2565 generic.go:356] "Generic (PLEG): container finished" podID="c8f1176d-5e04-4b50-ac50-8444754bcaf3" containerID="7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8" exitCode=0
Sep 16 18:55:26.930464 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.930365    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"
Sep 16 18:55:26.930464 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.930371    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn" event={"ID":"c8f1176d-5e04-4b50-ac50-8444754bcaf3","Type":"ContainerDied","Data":"7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8"}
Sep 16 18:55:26.930464 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.930409    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn" event={"ID":"c8f1176d-5e04-4b50-ac50-8444754bcaf3","Type":"ContainerDied","Data":"135a594a84d37bb1fd6c7d1dda623696605204530226af53e5d4bf39c4401ca2"}
Sep 16 18:55:26.930464 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.930430    2565 scope.go:117] "RemoveContainer" containerID="7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8"
Sep 16 18:55:26.939792 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.939775    2565 scope.go:117] "RemoveContainer" containerID="7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8"
Sep 16 18:55:26.940068 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:55:26.940045    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8\": container with ID starting with 7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8 not found: ID does not exist" containerID="7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8"
Sep 16 18:55:26.940173 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.940074    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8"} err="failed to get container status \"7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8\": rpc error: code = NotFound desc = could not find container \"7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8\": container with ID starting with 7dd54715f53b4ff5acc6a3d6f8c490ef4f8f4e44d4ef705f4d8a2c80a448b0d8 not found: ID does not exist"
Sep 16 18:55:26.946141 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.946099    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"]
Sep 16 18:55:26.950359 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.950337    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-hnmqn"]
Sep 16 18:55:26.982756 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.982699    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-m27t7"]
Sep 16 18:55:26.983006 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.982988    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f1176d-5e04-4b50-ac50-8444754bcaf3" containerName="seaweedfs-tls-custom"
Sep 16 18:55:26.983085 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.983015    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f1176d-5e04-4b50-ac50-8444754bcaf3" containerName="seaweedfs-tls-custom"
Sep 16 18:55:26.983198 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.983085    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f1176d-5e04-4b50-ac50-8444754bcaf3" containerName="seaweedfs-tls-custom"
Sep 16 18:55:26.987323 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.987304    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7"
Sep 16 18:55:26.990012 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.989996    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:55:26.990133 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.990117    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 16 18:55:26.990426 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.990411    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mv896\""
Sep 16 18:55:26.994312 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:26.994289    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-m27t7"]
Sep 16 18:55:27.128184 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.128159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/4af9067d-2ae7-4578-bb13-04e69a21e329-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-m27t7\" (UID: \"4af9067d-2ae7-4578-bb13-04e69a21e329\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7"
Sep 16 18:55:27.128279 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.128192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4af9067d-2ae7-4578-bb13-04e69a21e329-data\") pod \"seaweedfs-tls-custom-8b44d767d-m27t7\" (UID: \"4af9067d-2ae7-4578-bb13-04e69a21e329\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7"
Sep 16 18:55:27.128279 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.128222    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxjfq\" (UniqueName: \"kubernetes.io/projected/4af9067d-2ae7-4578-bb13-04e69a21e329-kube-api-access-sxjfq\") pod \"seaweedfs-tls-custom-8b44d767d-m27t7\" (UID: \"4af9067d-2ae7-4578-bb13-04e69a21e329\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7"
Sep 16 18:55:27.229515 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.229475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxjfq\" (UniqueName: \"kubernetes.io/projected/4af9067d-2ae7-4578-bb13-04e69a21e329-kube-api-access-sxjfq\") pod \"seaweedfs-tls-custom-8b44d767d-m27t7\" (UID: \"4af9067d-2ae7-4578-bb13-04e69a21e329\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7"
Sep 16 18:55:27.229615 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.229535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/4af9067d-2ae7-4578-bb13-04e69a21e329-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-m27t7\" (UID: \"4af9067d-2ae7-4578-bb13-04e69a21e329\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7"
Sep 16 18:55:27.229615 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.229561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4af9067d-2ae7-4578-bb13-04e69a21e329-data\") pod \"seaweedfs-tls-custom-8b44d767d-m27t7\" (UID: \"4af9067d-2ae7-4578-bb13-04e69a21e329\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7"
Sep 16 18:55:27.229902 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.229884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4af9067d-2ae7-4578-bb13-04e69a21e329-data\") pod \"seaweedfs-tls-custom-8b44d767d-m27t7\" (UID: \"4af9067d-2ae7-4578-bb13-04e69a21e329\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7"
Sep 16 18:55:27.232074 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.232042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/4af9067d-2ae7-4578-bb13-04e69a21e329-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-m27t7\" (UID: \"4af9067d-2ae7-4578-bb13-04e69a21e329\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7"
Sep 16 18:55:27.238167 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.238086    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxjfq\" (UniqueName: \"kubernetes.io/projected/4af9067d-2ae7-4578-bb13-04e69a21e329-kube-api-access-sxjfq\") pod \"seaweedfs-tls-custom-8b44d767d-m27t7\" (UID: \"4af9067d-2ae7-4578-bb13-04e69a21e329\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7"
Sep 16 18:55:27.296519 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.296488    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7"
Sep 16 18:55:27.568243 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.568160    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-m27t7"]
Sep 16 18:55:27.572157 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:55:27.572125    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af9067d_2ae7_4578_bb13_04e69a21e329.slice/crio-d0ee9b1fe5a7844669e9cb571fd1d464cd51c2d651664aeebc1512bf68b7725f WatchSource:0}: Error finding container d0ee9b1fe5a7844669e9cb571fd1d464cd51c2d651664aeebc1512bf68b7725f: Status 404 returned error can't find the container with id d0ee9b1fe5a7844669e9cb571fd1d464cd51c2d651664aeebc1512bf68b7725f
Sep 16 18:55:27.935526 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.935498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7" event={"ID":"4af9067d-2ae7-4578-bb13-04e69a21e329","Type":"ContainerStarted","Data":"7fc2bc392344774f9754e39834e2ab927570212a17ca0106b7a68de86082de21"}
Sep 16 18:55:27.935901 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.935547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7" event={"ID":"4af9067d-2ae7-4578-bb13-04e69a21e329","Type":"ContainerStarted","Data":"d0ee9b1fe5a7844669e9cb571fd1d464cd51c2d651664aeebc1512bf68b7725f"}
Sep 16 18:55:27.953566 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:27.953522    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-m27t7" podStartSLOduration=1.7064516859999999 podStartE2EDuration="1.953509045s" totalImagesPullingTime="247.057359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:27.952740076 +0000 UTC m=+611.766328934" watchObservedRunningTime="2026-09-16 18:55:27.953509045 +0000 UTC m=+611.767097903"
Sep 16 18:55:28.751974 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:28.751918    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8f1176d-5e04-4b50-ac50-8444754bcaf3" path="/var/lib/kubelet/pods/c8f1176d-5e04-4b50-ac50-8444754bcaf3/volumes"
Sep 16 18:55:39.705487 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.705419    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"]
Sep 16 18:55:39.709997 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.709982    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"
Sep 16 18:55:39.712541 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.712518    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 16 18:55:39.712659 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.712522    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 16 18:55:39.722062 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.722040    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"]
Sep 16 18:55:39.815647 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.815625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1b4495f8-ff25-4061-b0df-857669caf6b0-data\") pod \"seaweedfs-tls-serving-67b7477c6d-2vwxr\" (UID: \"1b4495f8-ff25-4061-b0df-857669caf6b0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"
Sep 16 18:55:39.815748 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.815661    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/1b4495f8-ff25-4061-b0df-857669caf6b0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-2vwxr\" (UID: \"1b4495f8-ff25-4061-b0df-857669caf6b0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"
Sep 16 18:55:39.815748 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.815691    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncbvj\" (UniqueName: \"kubernetes.io/projected/1b4495f8-ff25-4061-b0df-857669caf6b0-kube-api-access-ncbvj\") pod \"seaweedfs-tls-serving-67b7477c6d-2vwxr\" (UID: \"1b4495f8-ff25-4061-b0df-857669caf6b0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"
Sep 16 18:55:39.916415 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.916390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1b4495f8-ff25-4061-b0df-857669caf6b0-data\") pod \"seaweedfs-tls-serving-67b7477c6d-2vwxr\" (UID: \"1b4495f8-ff25-4061-b0df-857669caf6b0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"
Sep 16 18:55:39.916525 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.916434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/1b4495f8-ff25-4061-b0df-857669caf6b0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-2vwxr\" (UID: \"1b4495f8-ff25-4061-b0df-857669caf6b0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"
Sep 16 18:55:39.916525 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.916483    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncbvj\" (UniqueName: \"kubernetes.io/projected/1b4495f8-ff25-4061-b0df-857669caf6b0-kube-api-access-ncbvj\") pod \"seaweedfs-tls-serving-67b7477c6d-2vwxr\" (UID: \"1b4495f8-ff25-4061-b0df-857669caf6b0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"
Sep 16 18:55:39.916767 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.916746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1b4495f8-ff25-4061-b0df-857669caf6b0-data\") pod \"seaweedfs-tls-serving-67b7477c6d-2vwxr\" (UID: \"1b4495f8-ff25-4061-b0df-857669caf6b0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"
Sep 16 18:55:39.918950 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.918917    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/1b4495f8-ff25-4061-b0df-857669caf6b0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-2vwxr\" (UID: \"1b4495f8-ff25-4061-b0df-857669caf6b0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"
Sep 16 18:55:39.925360 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:39.925334    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncbvj\" (UniqueName: \"kubernetes.io/projected/1b4495f8-ff25-4061-b0df-857669caf6b0-kube-api-access-ncbvj\") pod \"seaweedfs-tls-serving-67b7477c6d-2vwxr\" (UID: \"1b4495f8-ff25-4061-b0df-857669caf6b0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"
Sep 16 18:55:40.017771 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:40.017713    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"
Sep 16 18:55:40.285524 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:40.285473    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr"]
Sep 16 18:55:40.287788 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:55:40.287753    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4495f8_ff25_4061_b0df_857669caf6b0.slice/crio-185b67a01804bc855e821644ec729d3c8763e49c0350512ad13f0e5482cfa8ca WatchSource:0}: Error finding container 185b67a01804bc855e821644ec729d3c8763e49c0350512ad13f0e5482cfa8ca: Status 404 returned error can't find the container with id 185b67a01804bc855e821644ec729d3c8763e49c0350512ad13f0e5482cfa8ca
Sep 16 18:55:40.985785 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:40.985746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr" event={"ID":"1b4495f8-ff25-4061-b0df-857669caf6b0","Type":"ContainerStarted","Data":"5a08ce058ac202ebcd3056dd3c160b2a3f8d0f463d522b416b3ef4b48d3dae80"}
Sep 16 18:55:40.986163 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:40.985803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr" event={"ID":"1b4495f8-ff25-4061-b0df-857669caf6b0","Type":"ContainerStarted","Data":"185b67a01804bc855e821644ec729d3c8763e49c0350512ad13f0e5482cfa8ca"}
Sep 16 18:55:41.002432 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:55:41.002384    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-2vwxr" podStartSLOduration=1.753833632 podStartE2EDuration="2.002368622s" totalImagesPullingTime="248.53499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:41.002264066 +0000 UTC m=+624.815852924" watchObservedRunningTime="2026-09-16 18:55:41.002368622 +0000 UTC m=+624.815957472"
Sep 16 18:56:07.992510 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:07.992473    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"]
Sep 16 18:56:07.995460 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:07.995437    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:07.998046 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:07.998017    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 16 18:56:07.998046 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:07.998039    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:56:07.998242 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:07.998029    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:56:07.998242 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:07.998091    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 16 18:56:07.999146 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:07.999125    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:56:07.999146 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:07.999138    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:56:07.999293 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:07.999231    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 18:56:08.009396 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.009354    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"]
Sep 16 18:56:08.093901 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.093875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.093984 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.093908    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c7320f-9996-42ec-aece-4ec84af5b3f9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.093984 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.093930    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg76c\" (UniqueName: \"kubernetes.io/projected/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kube-api-access-xg76c\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.094052 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.093998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.094052 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.094023    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c7320f-9996-42ec-aece-4ec84af5b3f9-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.194425 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.194395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.194517 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.194428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c7320f-9996-42ec-aece-4ec84af5b3f9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.194517 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.194446    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg76c\" (UniqueName: \"kubernetes.io/projected/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kube-api-access-xg76c\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.194589 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.194512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.194589 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.194545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c7320f-9996-42ec-aece-4ec84af5b3f9-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.194973 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.194952    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.195180 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.195162    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c7320f-9996-42ec-aece-4ec84af5b3f9-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.195230 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.195179    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.196800 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.196783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c7320f-9996-42ec-aece-4ec84af5b3f9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.202673 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.202655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg76c\" (UniqueName: \"kubernetes.io/projected/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kube-api-access-xg76c\") pod \"isvc-sklearn-batcher-predictor-7846bc9654-4gzdp\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.306280 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.306229    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:08.581662 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.581602    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"]
Sep 16 18:56:08.584778 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:56:08.584752    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c7320f_9996_42ec_aece_4ec84af5b3f9.slice/crio-4f0f61960207106b54f1b54f4add11701fe24044656db4c2fe240318828de418 WatchSource:0}: Error finding container 4f0f61960207106b54f1b54f4add11701fe24044656db4c2fe240318828de418: Status 404 returned error can't find the container with id 4f0f61960207106b54f1b54f4add11701fe24044656db4c2fe240318828de418
Sep 16 18:56:08.586918 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:08.586900    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:56:09.084520 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:09.084491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" event={"ID":"d0c7320f-9996-42ec-aece-4ec84af5b3f9","Type":"ContainerStarted","Data":"4f0f61960207106b54f1b54f4add11701fe24044656db4c2fe240318828de418"}
Sep 16 18:56:12.096784 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:12.096751    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" event={"ID":"d0c7320f-9996-42ec-aece-4ec84af5b3f9","Type":"ContainerStarted","Data":"e8859423e2dd3aa92cb15a2261593fa5cc5387865b6ba1e102fad1e0c77683a2"}
Sep 16 18:56:16.114536 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:16.114500    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerID="e8859423e2dd3aa92cb15a2261593fa5cc5387865b6ba1e102fad1e0c77683a2" exitCode=0
Sep 16 18:56:16.114862 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:16.114559    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" event={"ID":"d0c7320f-9996-42ec-aece-4ec84af5b3f9","Type":"ContainerDied","Data":"e8859423e2dd3aa92cb15a2261593fa5cc5387865b6ba1e102fad1e0c77683a2"}
Sep 16 18:56:29.177648 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:29.177619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" event={"ID":"d0c7320f-9996-42ec-aece-4ec84af5b3f9","Type":"ContainerStarted","Data":"f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb"}
Sep 16 18:56:31.187539 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:31.187472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" event={"ID":"d0c7320f-9996-42ec-aece-4ec84af5b3f9","Type":"ContainerStarted","Data":"d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0"}
Sep 16 18:56:34.201451 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:34.201408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" event={"ID":"d0c7320f-9996-42ec-aece-4ec84af5b3f9","Type":"ContainerStarted","Data":"5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a"}
Sep 16 18:56:34.201918 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:34.201506    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:34.201918 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:34.201668    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:34.202991 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:34.202967    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 16 18:56:34.225421 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:34.225377    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podStartSLOduration=2.894440205 podStartE2EDuration="27.225363757s" totalImagesPullingTime="20.429662521s" totalInitContainerRuntime="3.901261031s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:07 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:34.223505849 +0000 UTC m=+678.037094707" watchObservedRunningTime="2026-09-16 18:56:34.225363757 +0000 UTC m=+678.038952658"
Sep 16 18:56:35.205137 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:35.205116    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:35.205543 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:35.205231    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 16 18:56:35.206237 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:35.206210    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:35.209229 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:35.209209    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:56:36.208443 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:36.208402    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 16 18:56:36.208853 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:36.208825    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:37.211282 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:37.211243    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 16 18:56:37.211718 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:37.211687    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:47.211463 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:47.211422    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 16 18:56:47.211877 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:47.211855    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:57.211176 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:57.211133    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 16 18:56:57.211548 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:56:57.211519    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:07.211781 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:07.211736    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 16 18:57:07.212269 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:07.212243    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:17.212078 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:17.211976    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 16 18:57:17.212579 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:17.212552    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:27.212043 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:27.211993    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 16 18:57:27.212658 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:27.212626    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:37.211987 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:37.211942    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:57:37.212518 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:37.212492    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:57:42.777508 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.777471    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"]
Sep 16 18:57:42.777927 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.777822    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" containerID="cri-o://f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb" gracePeriod=30
Sep 16 18:57:42.777996 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.777912    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent" containerID="cri-o://5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a" gracePeriod=30
Sep 16 18:57:42.778073 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.777939    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kube-rbac-proxy" containerID="cri-o://d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0" gracePeriod=30
Sep 16 18:57:42.916668 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.916638    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"]
Sep 16 18:57:42.919884 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.919864    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:42.922480 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.922462    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 16 18:57:42.922676 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.922659    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 16 18:57:42.935421 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.935397    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"]
Sep 16 18:57:42.941211 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.941185    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:42.941292 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.941229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:42.941329 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.941304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzkbj\" (UniqueName: \"kubernetes.io/projected/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kube-api-access-fzkbj\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:42.941365 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.941335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:42.941397 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:42.941366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:43.042422 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.042365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:43.042422 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.042395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:43.042422 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.042421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzkbj\" (UniqueName: \"kubernetes.io/projected/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kube-api-access-fzkbj\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:43.042670 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.042651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:43.042716 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.042704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:43.042965 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.042948    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:43.043010 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.042990    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:43.043386 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.043366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:43.044826 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.044805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:43.050413 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.050390    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzkbj\" (UniqueName: \"kubernetes.io/projected/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kube-api-access-fzkbj\") pod \"isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:43.229710 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.229673    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:43.439966 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.439901    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerID="d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0" exitCode=2
Sep 16 18:57:43.440082 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.439972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" event={"ID":"d0c7320f-9996-42ec-aece-4ec84af5b3f9","Type":"ContainerDied","Data":"d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0"}
Sep 16 18:57:43.500752 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:43.500725    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"]
Sep 16 18:57:43.504339 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:57:43.504313    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5c0795_75a3_44ee_afa8_5ec417ae9aed.slice/crio-e23b9a47ab9ebd3656f76645995b68234aa4b87a9cfedd6260c8d2daf029a36f WatchSource:0}: Error finding container e23b9a47ab9ebd3656f76645995b68234aa4b87a9cfedd6260c8d2daf029a36f: Status 404 returned error can't find the container with id e23b9a47ab9ebd3656f76645995b68234aa4b87a9cfedd6260c8d2daf029a36f
Sep 16 18:57:44.445169 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:44.445133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" event={"ID":"1b5c0795-75a3-44ee-afa8-5ec417ae9aed","Type":"ContainerStarted","Data":"27b0c176bc1ab407b6d440fd57c313c569a8a02e4ab96043bdbfda547ec29ccd"}
Sep 16 18:57:44.445580 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:44.445233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" event={"ID":"1b5c0795-75a3-44ee-afa8-5ec417ae9aed","Type":"ContainerStarted","Data":"e23b9a47ab9ebd3656f76645995b68234aa4b87a9cfedd6260c8d2daf029a36f"}
Sep 16 18:57:45.205791 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:45.205748    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 16 18:57:47.212328 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:47.212289    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:47.212721 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:47.212533    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 16 18:57:47.459245 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:47.459218    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerID="f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb" exitCode=0
Sep 16 18:57:47.459342 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:47.459283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" event={"ID":"d0c7320f-9996-42ec-aece-4ec84af5b3f9","Type":"ContainerDied","Data":"f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb"}
Sep 16 18:57:48.465217 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:48.465188    2565 generic.go:356] "Generic (PLEG): container finished" podID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerID="27b0c176bc1ab407b6d440fd57c313c569a8a02e4ab96043bdbfda547ec29ccd" exitCode=0
Sep 16 18:57:48.465568 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:48.465246    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" event={"ID":"1b5c0795-75a3-44ee-afa8-5ec417ae9aed","Type":"ContainerDied","Data":"27b0c176bc1ab407b6d440fd57c313c569a8a02e4ab96043bdbfda547ec29ccd"}
Sep 16 18:57:49.471905 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:49.471867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" event={"ID":"1b5c0795-75a3-44ee-afa8-5ec417ae9aed","Type":"ContainerStarted","Data":"3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689"}
Sep 16 18:57:49.472283 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:49.471941    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" event={"ID":"1b5c0795-75a3-44ee-afa8-5ec417ae9aed","Type":"ContainerStarted","Data":"5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e"}
Sep 16 18:57:49.472283 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:49.472000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" event={"ID":"1b5c0795-75a3-44ee-afa8-5ec417ae9aed","Type":"ContainerStarted","Data":"101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80"}
Sep 16 18:57:49.472375 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:49.472317    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:49.472416 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:49.472387    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:49.472450 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:49.472424    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:49.473996 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:49.473961    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 16 18:57:49.474731 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:49.474694    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:49.496288 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:49.496241    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podStartSLOduration=3.446049934 podStartE2EDuration="7.496228721s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.050178787s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:49.494517479 +0000 UTC m=+753.308106338" watchObservedRunningTime="2026-09-16 18:57:49.496228721 +0000 UTC m=+753.309817574"
Sep 16 18:57:50.205472 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:50.205435    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 16 18:57:50.475841 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:50.475775    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 16 18:57:50.476153 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:50.476121    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:55.205375 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:55.205330    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 16 18:57:55.205790 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:55.205499    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:57:55.479839 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:55.479775    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:57:55.480344 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:55.480316    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 16 18:57:55.480656 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:55.480629    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:57.211976 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:57.211934    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:57.212493 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:57:57.212467    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 16 18:58:00.205762 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:00.205679    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 16 18:58:05.206252 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:05.206203    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 16 18:58:05.480293 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:05.480211    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 16 18:58:05.480649 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:05.480626    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:07.211749 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:07.211703    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:07.212147 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:07.211834    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:58:07.212941 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:07.212916    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 16 18:58:07.213051 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:07.213034    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:58:10.206228 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:10.206177    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 16 18:58:12.917622 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:12.917597    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:58:12.942065 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:12.942043    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c7320f-9996-42ec-aece-4ec84af5b3f9-proxy-tls\") pod \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") "
Sep 16 18:58:12.942178 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:12.942096    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg76c\" (UniqueName: \"kubernetes.io/projected/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kube-api-access-xg76c\") pod \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") "
Sep 16 18:58:12.942178 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:12.942175    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c7320f-9996-42ec-aece-4ec84af5b3f9-cabundle-cert\") pod \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") "
Sep 16 18:58:12.942264 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:12.942199    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kserve-provision-location\") pod \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") "
Sep 16 18:58:12.942264 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:12.942223    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kube-rbac-proxy-sar-config\") pod \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\" (UID: \"d0c7320f-9996-42ec-aece-4ec84af5b3f9\") "
Sep 16 18:58:12.942530 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:12.942503    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0c7320f-9996-42ec-aece-4ec84af5b3f9" (UID: "d0c7320f-9996-42ec-aece-4ec84af5b3f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:12.942647 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:12.942544    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c7320f-9996-42ec-aece-4ec84af5b3f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0c7320f-9996-42ec-aece-4ec84af5b3f9" (UID: "d0c7320f-9996-42ec-aece-4ec84af5b3f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:12.942731 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:12.942708    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0c7320f-9996-42ec-aece-4ec84af5b3f9" (UID: "d0c7320f-9996-42ec-aece-4ec84af5b3f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:12.944228 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:12.944203    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kube-api-access-xg76c" (OuterVolumeSpecName: "kube-api-access-xg76c") pod "d0c7320f-9996-42ec-aece-4ec84af5b3f9" (UID: "d0c7320f-9996-42ec-aece-4ec84af5b3f9"). InnerVolumeSpecName "kube-api-access-xg76c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:12.944703 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:12.944675    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c7320f-9996-42ec-aece-4ec84af5b3f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0c7320f-9996-42ec-aece-4ec84af5b3f9" (UID: "d0c7320f-9996-42ec-aece-4ec84af5b3f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:13.043187 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.043125    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c7320f-9996-42ec-aece-4ec84af5b3f9-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:13.043187 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.043146    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:13.043187 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.043156    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:13.043187 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.043164    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c7320f-9996-42ec-aece-4ec84af5b3f9-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:13.043187 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.043174    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg76c\" (UniqueName: \"kubernetes.io/projected/d0c7320f-9996-42ec-aece-4ec84af5b3f9-kube-api-access-xg76c\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:58:13.569770 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.569738    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerID="5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a" exitCode=0
Sep 16 18:58:13.569897 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.569812    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"
Sep 16 18:58:13.569897 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.569818    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" event={"ID":"d0c7320f-9996-42ec-aece-4ec84af5b3f9","Type":"ContainerDied","Data":"5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a"}
Sep 16 18:58:13.569973 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.569898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp" event={"ID":"d0c7320f-9996-42ec-aece-4ec84af5b3f9","Type":"ContainerDied","Data":"4f0f61960207106b54f1b54f4add11701fe24044656db4c2fe240318828de418"}
Sep 16 18:58:13.569973 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.569946    2565 scope.go:117] "RemoveContainer" containerID="5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a"
Sep 16 18:58:13.578378 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.578361    2565 scope.go:117] "RemoveContainer" containerID="d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0"
Sep 16 18:58:13.585537 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.585518    2565 scope.go:117] "RemoveContainer" containerID="f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb"
Sep 16 18:58:13.592385 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.592362    2565 scope.go:117] "RemoveContainer" containerID="e8859423e2dd3aa92cb15a2261593fa5cc5387865b6ba1e102fad1e0c77683a2"
Sep 16 18:58:13.596259 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.596238    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"]
Sep 16 18:58:13.599365 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.599347    2565 scope.go:117] "RemoveContainer" containerID="5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a"
Sep 16 18:58:13.599826 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:58:13.599803    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a\": container with ID starting with 5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a not found: ID does not exist" containerID="5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a"
Sep 16 18:58:13.599918 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.599837    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a"} err="failed to get container status \"5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a\": rpc error: code = NotFound desc = could not find container \"5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a\": container with ID starting with 5767c878b4dc1419ee770b60627093a7917223b9192026c52373c6fa862a083a not found: ID does not exist"
Sep 16 18:58:13.599918 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.599854    2565 scope.go:117] "RemoveContainer" containerID="d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0"
Sep 16 18:58:13.600173 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:58:13.600157    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0\": container with ID starting with d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0 not found: ID does not exist" containerID="d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0"
Sep 16 18:58:13.600275 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.600175    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0"} err="failed to get container status \"d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0\": rpc error: code = NotFound desc = could not find container \"d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0\": container with ID starting with d8fd93a09c8108afc84a4e9245349b294798d00282a6900bed6cddd9d90585d0 not found: ID does not exist"
Sep 16 18:58:13.600275 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.600188    2565 scope.go:117] "RemoveContainer" containerID="f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb"
Sep 16 18:58:13.600446 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:58:13.600423    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb\": container with ID starting with f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb not found: ID does not exist" containerID="f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb"
Sep 16 18:58:13.600505 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.600450    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb"} err="failed to get container status \"f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb\": rpc error: code = NotFound desc = could not find container \"f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb\": container with ID starting with f8e69578d245d5251771d7ee52f2393628c736fc4f1bd931575f40f67ecdc0fb not found: ID does not exist"
Sep 16 18:58:13.600505 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.600466    2565 scope.go:117] "RemoveContainer" containerID="e8859423e2dd3aa92cb15a2261593fa5cc5387865b6ba1e102fad1e0c77683a2"
Sep 16 18:58:13.600747 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:58:13.600729    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8859423e2dd3aa92cb15a2261593fa5cc5387865b6ba1e102fad1e0c77683a2\": container with ID starting with e8859423e2dd3aa92cb15a2261593fa5cc5387865b6ba1e102fad1e0c77683a2 not found: ID does not exist" containerID="e8859423e2dd3aa92cb15a2261593fa5cc5387865b6ba1e102fad1e0c77683a2"
Sep 16 18:58:13.600801 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.600751    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8859423e2dd3aa92cb15a2261593fa5cc5387865b6ba1e102fad1e0c77683a2"} err="failed to get container status \"e8859423e2dd3aa92cb15a2261593fa5cc5387865b6ba1e102fad1e0c77683a2\": rpc error: code = NotFound desc = could not find container \"e8859423e2dd3aa92cb15a2261593fa5cc5387865b6ba1e102fad1e0c77683a2\": container with ID starting with e8859423e2dd3aa92cb15a2261593fa5cc5387865b6ba1e102fad1e0c77683a2 not found: ID does not exist"
Sep 16 18:58:13.601002 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:13.600983    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7846bc9654-4gzdp"]
Sep 16 18:58:14.757236 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:14.757201    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" path="/var/lib/kubelet/pods/d0c7320f-9996-42ec-aece-4ec84af5b3f9/volumes"
Sep 16 18:58:15.480826 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:15.480788    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 16 18:58:15.481363 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:15.481335    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:25.480451 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:25.480409    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 16 18:58:25.480928 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:25.480901    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:35.480784 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:35.480737    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 16 18:58:35.481351 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:35.481320    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:45.480937 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:45.480834    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 16 18:58:45.481429 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:45.481375    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:55.481063 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:55.481014    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 16 18:58:55.481583 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:58:55.481493    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:05.481833 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:05.481797    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:59:05.482355 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:05.482328    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:59:18.002448 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:18.002413    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"]
Sep 16 18:59:18.002923 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:18.002670    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" containerID="cri-o://101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80" gracePeriod=30
Sep 16 18:59:18.002923 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:18.002707    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" containerID="cri-o://3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689" gracePeriod=30
Sep 16 18:59:18.002923 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:18.002759    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kube-rbac-proxy" containerID="cri-o://5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e" gracePeriod=30
Sep 16 18:59:18.808372 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:18.808344    2565 generic.go:356] "Generic (PLEG): container finished" podID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerID="5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e" exitCode=2
Sep 16 18:59:18.808531 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:18.808417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" event={"ID":"1b5c0795-75a3-44ee-afa8-5ec417ae9aed","Type":"ContainerDied","Data":"5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e"}
Sep 16 18:59:20.476208 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:20.476160    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 16 18:59:22.825118 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:22.825073    2565 generic.go:356] "Generic (PLEG): container finished" podID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerID="101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80" exitCode=0
Sep 16 18:59:22.825420 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:22.825140    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" event={"ID":"1b5c0795-75a3-44ee-afa8-5ec417ae9aed","Type":"ContainerDied","Data":"101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80"}
Sep 16 18:59:25.476408 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:25.476362    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 16 18:59:25.481123 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:25.481075    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:25.483402 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:25.483375    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 16 18:59:28.114200 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.114163    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"]
Sep 16 18:59:28.114563 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.114424    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container"
Sep 16 18:59:28.114563 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.114435    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container"
Sep 16 18:59:28.114563 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.114445    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="storage-initializer"
Sep 16 18:59:28.114563 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.114450    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="storage-initializer"
Sep 16 18:59:28.114563 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.114458    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kube-rbac-proxy"
Sep 16 18:59:28.114563 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.114464    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kube-rbac-proxy"
Sep 16 18:59:28.114563 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.114470    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent"
Sep 16 18:59:28.114563 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.114475    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent"
Sep 16 18:59:28.114563 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.114525    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kube-rbac-proxy"
Sep 16 18:59:28.114563 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.114534    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="kserve-container"
Sep 16 18:59:28.114563 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.114542    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c7320f-9996-42ec-aece-4ec84af5b3f9" containerName="agent"
Sep 16 18:59:28.117700 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.117681    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.120257 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.120231    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 16 18:59:28.120384 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.120309    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 16 18:59:28.140871 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.140846    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"]
Sep 16 18:59:28.238805 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.238778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2920713d-1049-4d97-b8c1-4cf7d54b75e3-cabundle-cert\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.238896 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.238818    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kserve-provision-location\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.238896 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.238847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.238966 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.238914    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9kcm\" (UniqueName: \"kubernetes.io/projected/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kube-api-access-p9kcm\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.238966 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.238950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2920713d-1049-4d97-b8c1-4cf7d54b75e3-proxy-tls\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.340223 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.340196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kserve-provision-location\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.340341 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.340228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.340341 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.340255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9kcm\" (UniqueName: \"kubernetes.io/projected/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kube-api-access-p9kcm\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.340341 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.340307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2920713d-1049-4d97-b8c1-4cf7d54b75e3-proxy-tls\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.340496 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.340363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2920713d-1049-4d97-b8c1-4cf7d54b75e3-cabundle-cert\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.340496 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:59:28.340373    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 16 18:59:28.340496 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:59:28.340435    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2920713d-1049-4d97-b8c1-4cf7d54b75e3-proxy-tls podName:2920713d-1049-4d97-b8c1-4cf7d54b75e3 nodeName:}" failed. No retries permitted until 2026-09-16 18:59:28.840414416 +0000 UTC m=+852.654003265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2920713d-1049-4d97-b8c1-4cf7d54b75e3-proxy-tls") pod "isvc-logger-predictor-5b98795569-x8t2b" (UID: "2920713d-1049-4d97-b8c1-4cf7d54b75e3") : secret "isvc-logger-predictor-serving-cert" not found
Sep 16 18:59:28.340645 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.340606    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kserve-provision-location\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.341006 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.340979    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.341088 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.340986    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2920713d-1049-4d97-b8c1-4cf7d54b75e3-cabundle-cert\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.350218 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.350197    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9kcm\" (UniqueName: \"kubernetes.io/projected/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kube-api-access-p9kcm\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.844614 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.844570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2920713d-1049-4d97-b8c1-4cf7d54b75e3-proxy-tls\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:28.847179 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:28.847148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2920713d-1049-4d97-b8c1-4cf7d54b75e3-proxy-tls\") pod \"isvc-logger-predictor-5b98795569-x8t2b\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:29.027870 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:29.027845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:29.303462 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:29.303428    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"]
Sep 16 18:59:29.305356 ip-10-0-135-44 kubenswrapper[2565]: W0916 18:59:29.305323    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2920713d_1049_4d97_b8c1_4cf7d54b75e3.slice/crio-7695f8a85a6a5e1d8e5015d87d2ec1636e726232f7c78cc6dff9f27434c75952 WatchSource:0}: Error finding container 7695f8a85a6a5e1d8e5015d87d2ec1636e726232f7c78cc6dff9f27434c75952: Status 404 returned error can't find the container with id 7695f8a85a6a5e1d8e5015d87d2ec1636e726232f7c78cc6dff9f27434c75952
Sep 16 18:59:29.852518 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:29.852482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" event={"ID":"2920713d-1049-4d97-b8c1-4cf7d54b75e3","Type":"ContainerStarted","Data":"0d1239536440a60c4e999959406bd520c83c484888d0bf59670238a6c012424f"}
Sep 16 18:59:29.852669 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:29.852566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" event={"ID":"2920713d-1049-4d97-b8c1-4cf7d54b75e3","Type":"ContainerStarted","Data":"7695f8a85a6a5e1d8e5015d87d2ec1636e726232f7c78cc6dff9f27434c75952"}
Sep 16 18:59:30.476826 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:30.476781    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 16 18:59:30.477221 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:30.476941    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:59:33.868988 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:33.868953    2565 generic.go:356] "Generic (PLEG): container finished" podID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerID="0d1239536440a60c4e999959406bd520c83c484888d0bf59670238a6c012424f" exitCode=0
Sep 16 18:59:33.869356 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:33.869005    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" event={"ID":"2920713d-1049-4d97-b8c1-4cf7d54b75e3","Type":"ContainerDied","Data":"0d1239536440a60c4e999959406bd520c83c484888d0bf59670238a6c012424f"}
Sep 16 18:59:34.875277 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:34.875234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" event={"ID":"2920713d-1049-4d97-b8c1-4cf7d54b75e3","Type":"ContainerStarted","Data":"268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a"}
Sep 16 18:59:34.875765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:34.875322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" event={"ID":"2920713d-1049-4d97-b8c1-4cf7d54b75e3","Type":"ContainerStarted","Data":"914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0"}
Sep 16 18:59:34.875765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:34.875362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" event={"ID":"2920713d-1049-4d97-b8c1-4cf7d54b75e3","Type":"ContainerStarted","Data":"de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef"}
Sep 16 18:59:34.875765 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:34.875732    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:34.875930 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:34.875862    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:34.877358 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:34.877321    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 18:59:34.899714 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:34.899670    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podStartSLOduration=3.086135739 podStartE2EDuration="6.899656657s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.813520918s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:34.898876456 +0000 UTC m=+858.712465312" watchObservedRunningTime="2026-09-16 18:59:34.899656657 +0000 UTC m=+858.713245510"
Sep 16 18:59:35.476004 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:35.475969    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 16 18:59:35.480732 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:35.480700    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:35.482975 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:35.482950    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 16 18:59:35.879381 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:35.879303    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 18:59:35.879381 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:35.879351    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:35.880375 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:35.880335    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:36.883768 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:36.883733    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 18:59:36.884238 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:36.884144    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:40.476401 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:40.476355    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 16 18:59:41.888314 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:41.888278    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 18:59:41.889026 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:41.888978    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 18:59:41.889232 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:41.889167    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:45.476853 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:45.476808    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 16 18:59:45.481419 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:45.481390    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:45.481541 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:45.481498    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:59:45.482613 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:45.482592    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 16 18:59:45.482709 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:45.482694    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:59:48.147807 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.147786    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:59:48.268967 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.268905    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-proxy-tls\") pod \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") "
Sep 16 18:59:48.268967 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.268949    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzkbj\" (UniqueName: \"kubernetes.io/projected/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kube-api-access-fzkbj\") pod \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") "
Sep 16 18:59:48.268967 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.268967    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-cabundle-cert\") pod \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") "
Sep 16 18:59:48.269216 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.268988    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kube-rbac-proxy-sar-config\") pod \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") "
Sep 16 18:59:48.269216 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.269131    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kserve-provision-location\") pod \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\" (UID: \"1b5c0795-75a3-44ee-afa8-5ec417ae9aed\") "
Sep 16 18:59:48.269405 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.269381    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b5c0795-75a3-44ee-afa8-5ec417ae9aed" (UID: "1b5c0795-75a3-44ee-afa8-5ec417ae9aed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:48.269460 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.269401    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1b5c0795-75a3-44ee-afa8-5ec417ae9aed" (UID: "1b5c0795-75a3-44ee-afa8-5ec417ae9aed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:48.269460 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.269411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b5c0795-75a3-44ee-afa8-5ec417ae9aed" (UID: "1b5c0795-75a3-44ee-afa8-5ec417ae9aed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:48.270945 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.270918    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b5c0795-75a3-44ee-afa8-5ec417ae9aed" (UID: "1b5c0795-75a3-44ee-afa8-5ec417ae9aed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:48.271058 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.270968    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kube-api-access-fzkbj" (OuterVolumeSpecName: "kube-api-access-fzkbj") pod "1b5c0795-75a3-44ee-afa8-5ec417ae9aed" (UID: "1b5c0795-75a3-44ee-afa8-5ec417ae9aed"). InnerVolumeSpecName "kube-api-access-fzkbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:48.370059 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.370035    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:59:48.370059 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.370055    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzkbj\" (UniqueName: \"kubernetes.io/projected/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kube-api-access-fzkbj\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:59:48.370202 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.370064    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:59:48.370202 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.370073    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:59:48.370202 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.370081    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b5c0795-75a3-44ee-afa8-5ec417ae9aed-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 18:59:48.932206 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.932177    2565 generic.go:356] "Generic (PLEG): container finished" podID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerID="3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689" exitCode=0
Sep 16 18:59:48.932383 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.932231    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"
Sep 16 18:59:48.932383 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.932255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" event={"ID":"1b5c0795-75a3-44ee-afa8-5ec417ae9aed","Type":"ContainerDied","Data":"3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689"}
Sep 16 18:59:48.932383 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.932329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb" event={"ID":"1b5c0795-75a3-44ee-afa8-5ec417ae9aed","Type":"ContainerDied","Data":"e23b9a47ab9ebd3656f76645995b68234aa4b87a9cfedd6260c8d2daf029a36f"}
Sep 16 18:59:48.932383 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.932377    2565 scope.go:117] "RemoveContainer" containerID="3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689"
Sep 16 18:59:48.940340 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.940320    2565 scope.go:117] "RemoveContainer" containerID="5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e"
Sep 16 18:59:48.947466 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.947449    2565 scope.go:117] "RemoveContainer" containerID="101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80"
Sep 16 18:59:48.954350 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.954313    2565 scope.go:117] "RemoveContainer" containerID="27b0c176bc1ab407b6d440fd57c313c569a8a02e4ab96043bdbfda547ec29ccd"
Sep 16 18:59:48.955910 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.955887    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"]
Sep 16 18:59:48.961372 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.961355    2565 scope.go:117] "RemoveContainer" containerID="3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689"
Sep 16 18:59:48.961664 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:59:48.961640    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689\": container with ID starting with 3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689 not found: ID does not exist" containerID="3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689"
Sep 16 18:59:48.961730 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.961672    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689"} err="failed to get container status \"3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689\": rpc error: code = NotFound desc = could not find container \"3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689\": container with ID starting with 3c74f660d58e3db53ef1fb94547d4d2a3c7cffcdbe331d3a7ffb478e69c16689 not found: ID does not exist"
Sep 16 18:59:48.961730 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.961695    2565 scope.go:117] "RemoveContainer" containerID="5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e"
Sep 16 18:59:48.961992 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:59:48.961975    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e\": container with ID starting with 5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e not found: ID does not exist" containerID="5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e"
Sep 16 18:59:48.962040 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.961998    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e"} err="failed to get container status \"5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e\": rpc error: code = NotFound desc = could not find container \"5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e\": container with ID starting with 5f3677f71298ae86cd5733559e11215cf53809aaced4df4e69689cfb7843201e not found: ID does not exist"
Sep 16 18:59:48.962040 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.962014    2565 scope.go:117] "RemoveContainer" containerID="101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80"
Sep 16 18:59:48.962291 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.962276    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c5f467b68-q6dwb"]
Sep 16 18:59:48.962345 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:59:48.962272    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80\": container with ID starting with 101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80 not found: ID does not exist" containerID="101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80"
Sep 16 18:59:48.962345 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.962322    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80"} err="failed to get container status \"101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80\": rpc error: code = NotFound desc = could not find container \"101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80\": container with ID starting with 101b3be8c642f9272c188c9d6bdd1a107414c8747a456a9719b996759e8efd80 not found: ID does not exist"
Sep 16 18:59:48.962345 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.962337    2565 scope.go:117] "RemoveContainer" containerID="27b0c176bc1ab407b6d440fd57c313c569a8a02e4ab96043bdbfda547ec29ccd"
Sep 16 18:59:48.962570 ip-10-0-135-44 kubenswrapper[2565]: E0916 18:59:48.962557    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b0c176bc1ab407b6d440fd57c313c569a8a02e4ab96043bdbfda547ec29ccd\": container with ID starting with 27b0c176bc1ab407b6d440fd57c313c569a8a02e4ab96043bdbfda547ec29ccd not found: ID does not exist" containerID="27b0c176bc1ab407b6d440fd57c313c569a8a02e4ab96043bdbfda547ec29ccd"
Sep 16 18:59:48.962607 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:48.962571    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b0c176bc1ab407b6d440fd57c313c569a8a02e4ab96043bdbfda547ec29ccd"} err="failed to get container status \"27b0c176bc1ab407b6d440fd57c313c569a8a02e4ab96043bdbfda547ec29ccd\": rpc error: code = NotFound desc = could not find container \"27b0c176bc1ab407b6d440fd57c313c569a8a02e4ab96043bdbfda547ec29ccd\": container with ID starting with 27b0c176bc1ab407b6d440fd57c313c569a8a02e4ab96043bdbfda547ec29ccd not found: ID does not exist"
Sep 16 18:59:50.751457 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:50.751419    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" path="/var/lib/kubelet/pods/1b5c0795-75a3-44ee-afa8-5ec417ae9aed/volumes"
Sep 16 18:59:51.889434 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:51.889396    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 18:59:51.889948 ip-10-0-135-44 kubenswrapper[2565]: I0916 18:59:51.889925    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:01.889030 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:01.888982    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 19:00:01.889571 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:01.889542    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:11.889504 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:11.889393    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 19:00:11.889893 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:11.889866    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:21.889696 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:21.889651    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 19:00:21.890299 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:21.890262    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:31.888866 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:31.888820    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 19:00:31.889396 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:31.889366    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:41.889590 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:41.889553    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 19:00:41.890183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:41.890156    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 19:00:53.303254 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:53.303223    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"]
Sep 16 19:00:53.303704 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:53.303488    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" containerID="cri-o://de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef" gracePeriod=30
Sep 16 19:00:53.303704 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:53.303499    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kube-rbac-proxy" containerID="cri-o://914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0" gracePeriod=30
Sep 16 19:00:53.303704 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:53.303517    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent" containerID="cri-o://268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a" gracePeriod=30
Sep 16 19:00:54.165823 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:54.165790    2565 generic.go:356] "Generic (PLEG): container finished" podID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerID="914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0" exitCode=2
Sep 16 19:00:54.165985 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:54.165836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" event={"ID":"2920713d-1049-4d97-b8c1-4cf7d54b75e3","Type":"ContainerDied","Data":"914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0"}
Sep 16 19:00:56.884300 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:56.884251    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 16 19:00:58.183982 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:58.183950    2565 generic.go:356] "Generic (PLEG): container finished" podID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerID="de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef" exitCode=0
Sep 16 19:00:58.184306 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:00:58.183990    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" event={"ID":"2920713d-1049-4d97-b8c1-4cf7d54b75e3","Type":"ContainerDied","Data":"de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef"}
Sep 16 19:01:01.884862 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:01.884824    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 16 19:01:01.889634 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:01.889603    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:01:01.890868 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:01.890838    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 19:01:06.884136 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:06.884077    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 16 19:01:06.884536 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:06.884220    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 19:01:11.884203 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:11.884157    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 16 19:01:11.889986 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:11.889952    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:01:11.890164 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:11.890137    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 19:01:16.884890 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:16.884841    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 16 19:01:21.884171 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:21.884128    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 16 19:01:21.889898 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:21.889857    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:01:21.890048 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:21.889983    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 19:01:21.890252 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:21.890229    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 19:01:21.890356 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:21.890341    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 19:01:23.440017 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.439998    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 19:01:23.510970 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.510946    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kserve-provision-location\") pod \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") "
Sep 16 19:01:23.511099 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.511009    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2920713d-1049-4d97-b8c1-4cf7d54b75e3-cabundle-cert\") pod \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") "
Sep 16 19:01:23.511099 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.511033    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9kcm\" (UniqueName: \"kubernetes.io/projected/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kube-api-access-p9kcm\") pod \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") "
Sep 16 19:01:23.511099 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.511060    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kube-rbac-proxy-sar-config\") pod \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") "
Sep 16 19:01:23.511099 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.511079    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2920713d-1049-4d97-b8c1-4cf7d54b75e3-proxy-tls\") pod \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\" (UID: \"2920713d-1049-4d97-b8c1-4cf7d54b75e3\") "
Sep 16 19:01:23.511331 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.511307    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2920713d-1049-4d97-b8c1-4cf7d54b75e3" (UID: "2920713d-1049-4d97-b8c1-4cf7d54b75e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:23.511589 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.511560    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2920713d-1049-4d97-b8c1-4cf7d54b75e3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2920713d-1049-4d97-b8c1-4cf7d54b75e3" (UID: "2920713d-1049-4d97-b8c1-4cf7d54b75e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:23.511722 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.511699    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2920713d-1049-4d97-b8c1-4cf7d54b75e3" (UID: "2920713d-1049-4d97-b8c1-4cf7d54b75e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:23.515301 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.515276    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kube-api-access-p9kcm" (OuterVolumeSpecName: "kube-api-access-p9kcm") pod "2920713d-1049-4d97-b8c1-4cf7d54b75e3" (UID: "2920713d-1049-4d97-b8c1-4cf7d54b75e3"). InnerVolumeSpecName "kube-api-access-p9kcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:23.515435 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.515416    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2920713d-1049-4d97-b8c1-4cf7d54b75e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2920713d-1049-4d97-b8c1-4cf7d54b75e3" (UID: "2920713d-1049-4d97-b8c1-4cf7d54b75e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:23.612293 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.612245    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:23.612293 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.612265    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2920713d-1049-4d97-b8c1-4cf7d54b75e3-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:23.612293 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.612276    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9kcm\" (UniqueName: \"kubernetes.io/projected/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kube-api-access-p9kcm\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:23.612293 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.612284    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2920713d-1049-4d97-b8c1-4cf7d54b75e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:23.612293 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:23.612292    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2920713d-1049-4d97-b8c1-4cf7d54b75e3-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:01:24.280651 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.280622    2565 generic.go:356] "Generic (PLEG): container finished" podID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerID="268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a" exitCode=0
Sep 16 19:01:24.280872 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.280670    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" event={"ID":"2920713d-1049-4d97-b8c1-4cf7d54b75e3","Type":"ContainerDied","Data":"268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a"}
Sep 16 19:01:24.280872 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.280705    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"
Sep 16 19:01:24.280872 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.280725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b" event={"ID":"2920713d-1049-4d97-b8c1-4cf7d54b75e3","Type":"ContainerDied","Data":"7695f8a85a6a5e1d8e5015d87d2ec1636e726232f7c78cc6dff9f27434c75952"}
Sep 16 19:01:24.280872 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.280757    2565 scope.go:117] "RemoveContainer" containerID="268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a"
Sep 16 19:01:24.289963 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.289944    2565 scope.go:117] "RemoveContainer" containerID="914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0"
Sep 16 19:01:24.297485 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.297468    2565 scope.go:117] "RemoveContainer" containerID="de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef"
Sep 16 19:01:24.304575 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.304556    2565 scope.go:117] "RemoveContainer" containerID="0d1239536440a60c4e999959406bd520c83c484888d0bf59670238a6c012424f"
Sep 16 19:01:24.305408 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.305358    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"]
Sep 16 19:01:24.307306 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.307280    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b98795569-x8t2b"]
Sep 16 19:01:24.311725 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.311611    2565 scope.go:117] "RemoveContainer" containerID="268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a"
Sep 16 19:01:24.311932 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:01:24.311910    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a\": container with ID starting with 268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a not found: ID does not exist" containerID="268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a"
Sep 16 19:01:24.311972 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.311940    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a"} err="failed to get container status \"268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a\": rpc error: code = NotFound desc = could not find container \"268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a\": container with ID starting with 268ba12e61e214cfbcc8d6144fa39b15117d7b3b310dcb9b4f22aa10fdc13e2a not found: ID does not exist"
Sep 16 19:01:24.311972 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.311956    2565 scope.go:117] "RemoveContainer" containerID="914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0"
Sep 16 19:01:24.312252 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:01:24.312234    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0\": container with ID starting with 914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0 not found: ID does not exist" containerID="914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0"
Sep 16 19:01:24.312305 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.312262    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0"} err="failed to get container status \"914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0\": rpc error: code = NotFound desc = could not find container \"914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0\": container with ID starting with 914aa1e0a8a5503f423a423076db894fba7ef3a8dba6b24cd34152af0d96d1c0 not found: ID does not exist"
Sep 16 19:01:24.312305 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.312281    2565 scope.go:117] "RemoveContainer" containerID="de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef"
Sep 16 19:01:24.312534 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:01:24.312515    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef\": container with ID starting with de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef not found: ID does not exist" containerID="de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef"
Sep 16 19:01:24.312600 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.312542    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef"} err="failed to get container status \"de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef\": rpc error: code = NotFound desc = could not find container \"de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef\": container with ID starting with de776a5cce20b5ef69a7b1dbe94c38e4b58557b58c7468082930d3d0458a88ef not found: ID does not exist"
Sep 16 19:01:24.312600 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.312563    2565 scope.go:117] "RemoveContainer" containerID="0d1239536440a60c4e999959406bd520c83c484888d0bf59670238a6c012424f"
Sep 16 19:01:24.312831 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:01:24.312809    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1239536440a60c4e999959406bd520c83c484888d0bf59670238a6c012424f\": container with ID starting with 0d1239536440a60c4e999959406bd520c83c484888d0bf59670238a6c012424f not found: ID does not exist" containerID="0d1239536440a60c4e999959406bd520c83c484888d0bf59670238a6c012424f"
Sep 16 19:01:24.312916 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.312831    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1239536440a60c4e999959406bd520c83c484888d0bf59670238a6c012424f"} err="failed to get container status \"0d1239536440a60c4e999959406bd520c83c484888d0bf59670238a6c012424f\": rpc error: code = NotFound desc = could not find container \"0d1239536440a60c4e999959406bd520c83c484888d0bf59670238a6c012424f\": container with ID starting with 0d1239536440a60c4e999959406bd520c83c484888d0bf59670238a6c012424f not found: ID does not exist"
Sep 16 19:01:24.751275 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:24.751252    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" path="/var/lib/kubelet/pods/2920713d-1049-4d97-b8c1-4cf7d54b75e3/volumes"
Sep 16 19:01:28.736917 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.736877    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"]
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737152    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737165    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737174    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="storage-initializer"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737180    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="storage-initializer"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737190    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737195    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737201    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kube-rbac-proxy"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737206    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kube-rbac-proxy"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737213    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="storage-initializer"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737218    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="storage-initializer"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737224    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737228    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737234    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737239    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737246    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kube-rbac-proxy"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737251    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kube-rbac-proxy"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737313    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kube-rbac-proxy"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737322    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="kserve-container"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737327    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kserve-container"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737334    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="agent"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737340    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b5c0795-75a3-44ee-afa8-5ec417ae9aed" containerName="agent"
Sep 16 19:01:28.737337 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.737345    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2920713d-1049-4d97-b8c1-4cf7d54b75e3" containerName="kube-rbac-proxy"
Sep 16 19:01:28.742209 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.742186    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.745336 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.745306    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:01:28.745471 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.745321    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:01:28.745471 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.745362    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 16 19:01:28.745471 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.745465    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:01:28.745732 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.745713    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:01:28.746586 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.746562    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:01:28.746694 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.746615    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:01:28.754321 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.754297    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"]
Sep 16 19:01:28.844331 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.844306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.844429 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.844338    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d947t\" (UniqueName: \"kubernetes.io/projected/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kube-api-access-d947t\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.844429 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.844366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.844429 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.844406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.844538 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.844439    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.945389 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.945359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.945504 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.945400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d947t\" (UniqueName: \"kubernetes.io/projected/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kube-api-access-d947t\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.945504 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.945431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.945504 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.945459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.945504 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.945490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.945699 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:01:28.945499    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 16 19:01:28.945699 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:01:28.945577    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-proxy-tls podName:08868969-8ca2-4bd4-a8c3-67a30ef5ce7a nodeName:}" failed. No retries permitted until 2026-09-16 19:01:29.445560403 +0000 UTC m=+973.259149238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" (UID: "08868969-8ca2-4bd4-a8c3-67a30ef5ce7a") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 16 19:01:28.945975 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.945948    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.946292 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.946271    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.946292 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.946283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:28.957292 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:28.957271    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d947t\" (UniqueName: \"kubernetes.io/projected/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kube-api-access-d947t\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:29.449030 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:29.449003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:29.451471 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:29.451438    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:29.653559 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:29.653535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:29.953434 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:01:29.953397    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08868969_8ca2_4bd4_a8c3_67a30ef5ce7a.slice/crio-31657a60b0e00a15cb1aa74df4abe38bbd109441f596cdffb959a37d6b5f5c74 WatchSource:0}: Error finding container 31657a60b0e00a15cb1aa74df4abe38bbd109441f596cdffb959a37d6b5f5c74: Status 404 returned error can't find the container with id 31657a60b0e00a15cb1aa74df4abe38bbd109441f596cdffb959a37d6b5f5c74
Sep 16 19:01:29.954000 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:29.953900    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"]
Sep 16 19:01:30.302574 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:30.302484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" event={"ID":"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a","Type":"ContainerStarted","Data":"738bfac2ef6da409ff519cdde82c9d26bc27d25ba3f19ed18ae74eac20ff3a32"}
Sep 16 19:01:30.302697 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:30.302596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" event={"ID":"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a","Type":"ContainerStarted","Data":"31657a60b0e00a15cb1aa74df4abe38bbd109441f596cdffb959a37d6b5f5c74"}
Sep 16 19:01:34.318439 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:34.318405    2565 generic.go:356] "Generic (PLEG): container finished" podID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerID="738bfac2ef6da409ff519cdde82c9d26bc27d25ba3f19ed18ae74eac20ff3a32" exitCode=0
Sep 16 19:01:34.318752 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:34.318484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" event={"ID":"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a","Type":"ContainerDied","Data":"738bfac2ef6da409ff519cdde82c9d26bc27d25ba3f19ed18ae74eac20ff3a32"}
Sep 16 19:01:34.320032 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:34.320015    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:01:56.418001 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:56.417959    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" event={"ID":"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a","Type":"ContainerStarted","Data":"edef02bc85f64a735d03211905ac1384aeebe1a7f88e273045bcdb6580938155"}
Sep 16 19:01:56.418418 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:56.418033    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" event={"ID":"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a","Type":"ContainerStarted","Data":"831970dbcc004eaa6da5392a0d3bdda33aec1515ba5b1e473e3539b6632e9ae6"}
Sep 16 19:01:56.418418 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:56.418272    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:01:56.441012 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:56.440963    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" podStartSLOduration=3.504136624 podStartE2EDuration="28.44094679s" totalImagesPullingTime="21.124058625s" totalInitContainerRuntime="3.812751541s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:56.439206646 +0000 UTC m=+1000.252795498" watchObservedRunningTime="2026-09-16 19:01:56.44094679 +0000 UTC m=+1000.254535648"
Sep 16 19:01:57.422249 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:01:57.422217    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:02:03.431262 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:02:03.431230    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:02:33.432872 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:02:33.432833    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.22:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 16 19:02:43.432657 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:02:43.432614    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.22:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 16 19:02:53.432744 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:02:53.432705    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.22:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 16 19:03:03.448810 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:03.448770    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:03:08.840447 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:08.840409    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"]
Sep 16 19:03:08.840920 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:08.840638    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="kserve-container" containerID="cri-o://831970dbcc004eaa6da5392a0d3bdda33aec1515ba5b1e473e3539b6632e9ae6" gracePeriod=30
Sep 16 19:03:08.840920 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:08.840694    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="kube-rbac-proxy" containerID="cri-o://edef02bc85f64a735d03211905ac1384aeebe1a7f88e273045bcdb6580938155" gracePeriod=30
Sep 16 19:03:09.677620 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:09.677539    2565 generic.go:356] "Generic (PLEG): container finished" podID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerID="edef02bc85f64a735d03211905ac1384aeebe1a7f88e273045bcdb6580938155" exitCode=2
Sep 16 19:03:09.677620 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:09.677581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" event={"ID":"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a","Type":"ContainerDied","Data":"edef02bc85f64a735d03211905ac1384aeebe1a7f88e273045bcdb6580938155"}
Sep 16 19:03:12.693255 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.693224    2565 generic.go:356] "Generic (PLEG): container finished" podID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerID="831970dbcc004eaa6da5392a0d3bdda33aec1515ba5b1e473e3539b6632e9ae6" exitCode=0
Sep 16 19:03:12.693255 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.693264    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" event={"ID":"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a","Type":"ContainerDied","Data":"831970dbcc004eaa6da5392a0d3bdda33aec1515ba5b1e473e3539b6632e9ae6"}
Sep 16 19:03:12.786641 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.786621    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:03:12.864990 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.864937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d947t\" (UniqueName: \"kubernetes.io/projected/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kube-api-access-d947t\") pod \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") "
Sep 16 19:03:12.865078 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.864994    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-proxy-tls\") pod \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") "
Sep 16 19:03:12.865078 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.865013    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kube-rbac-proxy-sar-config\") pod \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") "
Sep 16 19:03:12.865078 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.865028    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-cabundle-cert\") pod \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") "
Sep 16 19:03:12.865078 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.865049    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kserve-provision-location\") pod \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\" (UID: \"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a\") "
Sep 16 19:03:12.865412 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.865357    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" (UID: "08868969-8ca2-4bd4-a8c3-67a30ef5ce7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:12.865523 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.865440    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" (UID: "08868969-8ca2-4bd4-a8c3-67a30ef5ce7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:12.865523 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.865443    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" (UID: "08868969-8ca2-4bd4-a8c3-67a30ef5ce7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:12.866917 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.866894    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" (UID: "08868969-8ca2-4bd4-a8c3-67a30ef5ce7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:12.866990 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.866944    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kube-api-access-d947t" (OuterVolumeSpecName: "kube-api-access-d947t") pod "08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" (UID: "08868969-8ca2-4bd4-a8c3-67a30ef5ce7a"). InnerVolumeSpecName "kube-api-access-d947t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:12.965493 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.965470    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:03:12.965493 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.965491    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:03:12.965595 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.965499    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:03:12.965595 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.965508    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:03:12.965595 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:12.965516    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d947t\" (UniqueName: \"kubernetes.io/projected/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a-kube-api-access-d947t\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:03:13.698954 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:13.698928    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"
Sep 16 19:03:13.699292 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:13.698942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs" event={"ID":"08868969-8ca2-4bd4-a8c3-67a30ef5ce7a","Type":"ContainerDied","Data":"31657a60b0e00a15cb1aa74df4abe38bbd109441f596cdffb959a37d6b5f5c74"}
Sep 16 19:03:13.699292 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:13.699040    2565 scope.go:117] "RemoveContainer" containerID="edef02bc85f64a735d03211905ac1384aeebe1a7f88e273045bcdb6580938155"
Sep 16 19:03:13.707468 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:13.707448    2565 scope.go:117] "RemoveContainer" containerID="831970dbcc004eaa6da5392a0d3bdda33aec1515ba5b1e473e3539b6632e9ae6"
Sep 16 19:03:13.714816 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:13.714795    2565 scope.go:117] "RemoveContainer" containerID="738bfac2ef6da409ff519cdde82c9d26bc27d25ba3f19ed18ae74eac20ff3a32"
Sep 16 19:03:13.724059 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:13.724035    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"]
Sep 16 19:03:13.728594 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:13.728572    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lffcs"]
Sep 16 19:03:14.034444 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.034415    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"]
Sep 16 19:03:14.034697 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.034681    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="storage-initializer"
Sep 16 19:03:14.034755 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.034698    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="storage-initializer"
Sep 16 19:03:14.034755 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.034719    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="kserve-container"
Sep 16 19:03:14.034755 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.034725    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="kserve-container"
Sep 16 19:03:14.034755 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.034731    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="kube-rbac-proxy"
Sep 16 19:03:14.034755 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.034736    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="kube-rbac-proxy"
Sep 16 19:03:14.034923 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.034778    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="kube-rbac-proxy"
Sep 16 19:03:14.034923 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.034787    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" containerName="kserve-container"
Sep 16 19:03:14.039236 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.039218    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.042400 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.042378    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:03:14.042509 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.042481    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:03:14.042583 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.042505    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 16 19:03:14.042583 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.042517    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:03:14.042583 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.042504    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:03:14.042583 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.042557    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:14.042772 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.042623    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:03:14.050587 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.050562    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"]
Sep 16 19:03:14.173768 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.173738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.173768 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.173768    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.173933 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.173802    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htn4x\" (UniqueName: \"kubernetes.io/projected/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kube-api-access-htn4x\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.173933 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.173889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.173933 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.173912    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.274591 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.274567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.274706 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.274595    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.274706 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.274636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htn4x\" (UniqueName: \"kubernetes.io/projected/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kube-api-access-htn4x\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.274706 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.274668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.274847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.274825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.275288 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.275263    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.275453 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.275427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.275508 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.275437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.277402 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.277380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.286432 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.286367    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htn4x\" (UniqueName: \"kubernetes.io/projected/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kube-api-access-htn4x\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.349433 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.349406    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:14.624638 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.624613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"]
Sep 16 19:03:14.626619 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:03:14.626592    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bd4f1c_6e9c_4bc4_9c02_dc2dcf1ed08b.slice/crio-8006f2925c55974c722b131f6ec85b54e50c067519dcfc9a6b0628f456d23d6a WatchSource:0}: Error finding container 8006f2925c55974c722b131f6ec85b54e50c067519dcfc9a6b0628f456d23d6a: Status 404 returned error can't find the container with id 8006f2925c55974c722b131f6ec85b54e50c067519dcfc9a6b0628f456d23d6a
Sep 16 19:03:14.704317 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.704288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" event={"ID":"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b","Type":"ContainerStarted","Data":"ce59263f5c4030ca0c23eec2138728cb7951036a88c387964034fdb948ef8638"}
Sep 16 19:03:14.704619 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.704359    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" event={"ID":"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b","Type":"ContainerStarted","Data":"8006f2925c55974c722b131f6ec85b54e50c067519dcfc9a6b0628f456d23d6a"}
Sep 16 19:03:14.752785 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:14.752758    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08868969-8ca2-4bd4-a8c3-67a30ef5ce7a" path="/var/lib/kubelet/pods/08868969-8ca2-4bd4-a8c3-67a30ef5ce7a/volumes"
Sep 16 19:03:18.720930 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:18.720893    2565 generic.go:356] "Generic (PLEG): container finished" podID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerID="ce59263f5c4030ca0c23eec2138728cb7951036a88c387964034fdb948ef8638" exitCode=0
Sep 16 19:03:18.721311 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:18.720976    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" event={"ID":"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b","Type":"ContainerDied","Data":"ce59263f5c4030ca0c23eec2138728cb7951036a88c387964034fdb948ef8638"}
Sep 16 19:03:19.727463 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:19.727430    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" event={"ID":"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b","Type":"ContainerStarted","Data":"ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16"}
Sep 16 19:03:19.727803 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:19.727488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" event={"ID":"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b","Type":"ContainerStarted","Data":"ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134"}
Sep 16 19:03:19.727803 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:19.727684    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:19.751087 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:19.751044    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" podStartSLOduration=1.912314105 podStartE2EDuration="5.751032421s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.838718316s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:19.749724754 +0000 UTC m=+1083.563313611" watchObservedRunningTime="2026-09-16 19:03:19.751032421 +0000 UTC m=+1083.564621277"
Sep 16 19:03:20.733442 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:20.733410    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:26.744798 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:26.744761    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:03:56.745695 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:03:56.745653    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.23:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 16 19:04:06.745848 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:06.745793    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.23:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 16 19:04:16.745398 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:16.745360    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.23:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 16 19:04:26.851933 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:26.851902    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:04:34.095227 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:34.095191    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"]
Sep 16 19:04:34.095700 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:34.095418    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kserve-container" containerID="cri-o://ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134" gracePeriod=30
Sep 16 19:04:34.095700 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:34.095472    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kube-rbac-proxy" containerID="cri-o://ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16" gracePeriod=30
Sep 16 19:04:34.999408 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:34.999374    2565 generic.go:356] "Generic (PLEG): container finished" podID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerID="ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16" exitCode=2
Sep 16 19:04:34.999568 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:34.999431    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" event={"ID":"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b","Type":"ContainerDied","Data":"ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16"}
Sep 16 19:04:36.737900 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:36.737848    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 16 19:04:36.746180 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:36.746145    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.23:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 16 19:04:38.230307 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.230286    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:04:38.336386 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.336334    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-proxy-tls\") pod \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") "
Sep 16 19:04:38.336386 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.336361    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kube-rbac-proxy-sar-config\") pod \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") "
Sep 16 19:04:38.336386 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.336378    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kserve-provision-location\") pod \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") "
Sep 16 19:04:38.336552 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.336408    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-cabundle-cert\") pod \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") "
Sep 16 19:04:38.336552 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.336426    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htn4x\" (UniqueName: \"kubernetes.io/projected/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kube-api-access-htn4x\") pod \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\" (UID: \"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b\") "
Sep 16 19:04:38.336645 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.336622    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" (UID: "84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:38.336764 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.336741    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" (UID: "84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:38.336909 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.336888    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" (UID: "84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:38.338389 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.338365    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" (UID: "84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:38.338450 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.338392    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kube-api-access-htn4x" (OuterVolumeSpecName: "kube-api-access-htn4x") pod "84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" (UID: "84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b"). InnerVolumeSpecName "kube-api-access-htn4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:38.436944 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.436922    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:38.436944 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.436942    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htn4x\" (UniqueName: \"kubernetes.io/projected/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kube-api-access-htn4x\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:38.437059 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.436953    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:38.437059 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.436962    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:38.437059 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:38.436969    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:04:39.015505 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.015437    2565 generic.go:356] "Generic (PLEG): container finished" podID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerID="ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134" exitCode=0
Sep 16 19:04:39.015505 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.015499    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"
Sep 16 19:04:39.015643 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.015520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" event={"ID":"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b","Type":"ContainerDied","Data":"ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134"}
Sep 16 19:04:39.015643 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.015595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh" event={"ID":"84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b","Type":"ContainerDied","Data":"8006f2925c55974c722b131f6ec85b54e50c067519dcfc9a6b0628f456d23d6a"}
Sep 16 19:04:39.015643 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.015624    2565 scope.go:117] "RemoveContainer" containerID="ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16"
Sep 16 19:04:39.023926 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.023902    2565 scope.go:117] "RemoveContainer" containerID="ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134"
Sep 16 19:04:39.031304 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.031278    2565 scope.go:117] "RemoveContainer" containerID="ce59263f5c4030ca0c23eec2138728cb7951036a88c387964034fdb948ef8638"
Sep 16 19:04:39.036121 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.036083    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"]
Sep 16 19:04:39.038834 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.038818    2565 scope.go:117] "RemoveContainer" containerID="ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16"
Sep 16 19:04:39.039150 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:04:39.039124    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16\": container with ID starting with ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16 not found: ID does not exist" containerID="ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16"
Sep 16 19:04:39.039247 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.039152    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16"} err="failed to get container status \"ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16\": rpc error: code = NotFound desc = could not find container \"ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16\": container with ID starting with ec0fd1a7eab7ca664651100f9779210517f9648f7023ca71e9b41c7d00c4fc16 not found: ID does not exist"
Sep 16 19:04:39.039247 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.039166    2565 scope.go:117] "RemoveContainer" containerID="ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134"
Sep 16 19:04:39.039438 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:04:39.039420    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134\": container with ID starting with ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134 not found: ID does not exist" containerID="ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134"
Sep 16 19:04:39.039482 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.039443    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134"} err="failed to get container status \"ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134\": rpc error: code = NotFound desc = could not find container \"ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134\": container with ID starting with ed4304bce72d1fac82599087d3ac90cb472b0462a8ad27785c33f512de298134 not found: ID does not exist"
Sep 16 19:04:39.039482 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.039458    2565 scope.go:117] "RemoveContainer" containerID="ce59263f5c4030ca0c23eec2138728cb7951036a88c387964034fdb948ef8638"
Sep 16 19:04:39.039697 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:04:39.039678    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce59263f5c4030ca0c23eec2138728cb7951036a88c387964034fdb948ef8638\": container with ID starting with ce59263f5c4030ca0c23eec2138728cb7951036a88c387964034fdb948ef8638 not found: ID does not exist" containerID="ce59263f5c4030ca0c23eec2138728cb7951036a88c387964034fdb948ef8638"
Sep 16 19:04:39.039737 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.039701    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce59263f5c4030ca0c23eec2138728cb7951036a88c387964034fdb948ef8638"} err="failed to get container status \"ce59263f5c4030ca0c23eec2138728cb7951036a88c387964034fdb948ef8638\": rpc error: code = NotFound desc = could not find container \"ce59263f5c4030ca0c23eec2138728cb7951036a88c387964034fdb948ef8638\": container with ID starting with ce59263f5c4030ca0c23eec2138728cb7951036a88c387964034fdb948ef8638 not found: ID does not exist"
Sep 16 19:04:39.044391 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.044371    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zcpxh"]
Sep 16 19:04:39.271408 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.271353    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"]
Sep 16 19:04:39.271673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.271587    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kserve-container"
Sep 16 19:04:39.271673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.271598    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kserve-container"
Sep 16 19:04:39.271673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.271610    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kube-rbac-proxy"
Sep 16 19:04:39.271673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.271615    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kube-rbac-proxy"
Sep 16 19:04:39.271673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.271623    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="storage-initializer"
Sep 16 19:04:39.271673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.271629    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="storage-initializer"
Sep 16 19:04:39.271673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.271667    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kserve-container"
Sep 16 19:04:39.271673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.271675    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" containerName="kube-rbac-proxy"
Sep 16 19:04:39.276028 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.276011    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.279986 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.279961    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:04:39.280088 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.279992    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:04:39.280088 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.280007    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:04:39.280381 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.280360    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 16 19:04:39.280477 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.280386    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:04:39.280747 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.280732    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:04:39.280802 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.280766    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:04:39.291976 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.291956    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"]
Sep 16 19:04:39.444405 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.444377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.444517 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.444417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.444517 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.444444    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.444517 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.444512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.444628 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.444590    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2nn\" (UniqueName: \"kubernetes.io/projected/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kube-api-access-vx2nn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.545399 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.545336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.545399 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.545367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.545553 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.545403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.546142 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.545457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2nn\" (UniqueName: \"kubernetes.io/projected/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kube-api-access-vx2nn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.546755 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.546678    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.546755 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.546713    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.546937 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.546764    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.546937 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.546838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.552387 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.548728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.556334 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.556307    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2nn\" (UniqueName: \"kubernetes.io/projected/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kube-api-access-vx2nn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.585278 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.585250    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:39.858157 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:39.858132    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"]
Sep 16 19:04:39.859999 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:04:39.859968    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8268f8a6_2d8d_43b7_a1cc_58ca9648ae8c.slice/crio-53f30ee5a0a3f44c2dc2b21cb06e8b596d465e6d36bded04657e1fad3222740d WatchSource:0}: Error finding container 53f30ee5a0a3f44c2dc2b21cb06e8b596d465e6d36bded04657e1fad3222740d: Status 404 returned error can't find the container with id 53f30ee5a0a3f44c2dc2b21cb06e8b596d465e6d36bded04657e1fad3222740d
Sep 16 19:04:40.021196 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:40.021169    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" event={"ID":"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c","Type":"ContainerStarted","Data":"4460685f94787307cf8c4fd7a7056002d0922776f1e636d0b4f42784e09ac14d"}
Sep 16 19:04:40.021285 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:40.021217    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" event={"ID":"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c","Type":"ContainerStarted","Data":"53f30ee5a0a3f44c2dc2b21cb06e8b596d465e6d36bded04657e1fad3222740d"}
Sep 16 19:04:40.751474 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:40.751446    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b" path="/var/lib/kubelet/pods/84bd4f1c-6e9c-4bc4-9c02-dc2dcf1ed08b/volumes"
Sep 16 19:04:44.037790 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:44.037723    2565 generic.go:356] "Generic (PLEG): container finished" podID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerID="4460685f94787307cf8c4fd7a7056002d0922776f1e636d0b4f42784e09ac14d" exitCode=0
Sep 16 19:04:44.037790 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:44.037759    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" event={"ID":"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c","Type":"ContainerDied","Data":"4460685f94787307cf8c4fd7a7056002d0922776f1e636d0b4f42784e09ac14d"}
Sep 16 19:04:45.043927 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:45.043885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" event={"ID":"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c","Type":"ContainerStarted","Data":"7e51248cba330af3ecb4a566a47eb41783b87ec4e507f53d881186b3ca7534e4"}
Sep 16 19:04:45.044443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:45.043962    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" event={"ID":"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c","Type":"ContainerStarted","Data":"2602094c3eb8793adfb68d254f2df08defd99c3257882fb310b172cd28b80cca"}
Sep 16 19:04:45.044443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:45.044221    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:45.080588 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:45.080542    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" podStartSLOduration=2.259813612 podStartE2EDuration="6.080529831s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.820716219s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:45.07902809 +0000 UTC m=+1168.892616946" watchObservedRunningTime="2026-09-16 19:04:45.080529831 +0000 UTC m=+1168.894118687"
Sep 16 19:04:46.048467 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:46.048440    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:04:52.057596 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:04:52.057563    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:05:22.058990 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:05:22.058944    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.24:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 16 19:05:32.058465 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:05:32.058421    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.24:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 16 19:05:42.058642 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:05:42.058598    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.24:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 16 19:05:52.149646 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:05:52.149608    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:05:59.362149 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:05:59.362086    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"]
Sep 16 19:05:59.362600 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:05:59.362324    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kserve-container" containerID="cri-o://2602094c3eb8793adfb68d254f2df08defd99c3257882fb310b172cd28b80cca" gracePeriod=30
Sep 16 19:05:59.362600 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:05:59.362391    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kube-rbac-proxy" containerID="cri-o://7e51248cba330af3ecb4a566a47eb41783b87ec4e507f53d881186b3ca7534e4" gracePeriod=30
Sep 16 19:06:00.318327 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:00.318289    2565 generic.go:356] "Generic (PLEG): container finished" podID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerID="7e51248cba330af3ecb4a566a47eb41783b87ec4e507f53d881186b3ca7534e4" exitCode=2
Sep 16 19:06:00.318327 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:00.318333    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" event={"ID":"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c","Type":"ContainerDied","Data":"7e51248cba330af3ecb4a566a47eb41783b87ec4e507f53d881186b3ca7534e4"}
Sep 16 19:06:02.052781 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:02.052738    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 16 19:06:02.058268 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:02.058236    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.24:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 16 19:06:03.331187 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.331155    2565 generic.go:356] "Generic (PLEG): container finished" podID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerID="2602094c3eb8793adfb68d254f2df08defd99c3257882fb310b172cd28b80cca" exitCode=0
Sep 16 19:06:03.331528 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.331228    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" event={"ID":"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c","Type":"ContainerDied","Data":"2602094c3eb8793adfb68d254f2df08defd99c3257882fb310b172cd28b80cca"}
Sep 16 19:06:03.596661 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.596637    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:06:03.701407 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.701382    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kserve-provision-location\") pod \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") "
Sep 16 19:06:03.701512 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.701411    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kube-rbac-proxy-sar-config\") pod \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") "
Sep 16 19:06:03.701512 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.701434    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx2nn\" (UniqueName: \"kubernetes.io/projected/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kube-api-access-vx2nn\") pod \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") "
Sep 16 19:06:03.701512 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.701470    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-proxy-tls\") pod \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") "
Sep 16 19:06:03.701654 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.701591    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-cabundle-cert\") pod \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\" (UID: \"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c\") "
Sep 16 19:06:03.701748 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.701722    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" (UID: "8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:06:03.701748 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.701735    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" (UID: "8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:03.701864 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.701847    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:06:03.701913 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.701868    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:06:03.701968 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.701904    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" (UID: "8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:06:03.703439 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.703419    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" (UID: "8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:03.703527 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.703484    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kube-api-access-vx2nn" (OuterVolumeSpecName: "kube-api-access-vx2nn") pod "8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" (UID: "8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c"). InnerVolumeSpecName "kube-api-access-vx2nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:03.802496 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.802441    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx2nn\" (UniqueName: \"kubernetes.io/projected/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-kube-api-access-vx2nn\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:06:03.802496 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.802464    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:06:03.802496 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:03.802475    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:06:04.336846 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:04.336816    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt" event={"ID":"8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c","Type":"ContainerDied","Data":"53f30ee5a0a3f44c2dc2b21cb06e8b596d465e6d36bded04657e1fad3222740d"}
Sep 16 19:06:04.336846 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:04.336844    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"
Sep 16 19:06:04.337280 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:04.336882    2565 scope.go:117] "RemoveContainer" containerID="7e51248cba330af3ecb4a566a47eb41783b87ec4e507f53d881186b3ca7534e4"
Sep 16 19:06:04.345521 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:04.345501    2565 scope.go:117] "RemoveContainer" containerID="2602094c3eb8793adfb68d254f2df08defd99c3257882fb310b172cd28b80cca"
Sep 16 19:06:04.352863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:04.352844    2565 scope.go:117] "RemoveContainer" containerID="4460685f94787307cf8c4fd7a7056002d0922776f1e636d0b4f42784e09ac14d"
Sep 16 19:06:04.359044 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:04.359019    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"]
Sep 16 19:06:04.362890 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:04.362863    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-hcvnt"]
Sep 16 19:06:04.751435 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:04.751406    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" path="/var/lib/kubelet/pods/8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c/volumes"
Sep 16 19:06:38.187119 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.187073    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"]
Sep 16 19:06:38.187591 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.187407    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="storage-initializer"
Sep 16 19:06:38.187591 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.187421    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="storage-initializer"
Sep 16 19:06:38.187591 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.187432    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kserve-container"
Sep 16 19:06:38.187591 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.187438    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kserve-container"
Sep 16 19:06:38.187591 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.187449    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kube-rbac-proxy"
Sep 16 19:06:38.187591 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.187454    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kube-rbac-proxy"
Sep 16 19:06:38.187591 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.187504    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kube-rbac-proxy"
Sep 16 19:06:38.187591 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.187511    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8268f8a6-2d8d-43b7-a1cc-58ca9648ae8c" containerName="kserve-container"
Sep 16 19:06:38.191736 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.191717    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.194565 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.194536    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:06:38.194565 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.194555    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:06:38.194783 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.194611    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:06:38.194783 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.194682    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 16 19:06:38.194783 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.194700    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:06:38.194929 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.194864    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:06:38.195525 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.195508    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 16 19:06:38.201074 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.201048    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"]
Sep 16 19:06:38.322970 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.322942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kserve-provision-location\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.323143 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.322980    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b09e6726-a6c8-4582-aef8-02149fb5bd2e-cabundle-cert\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.323143 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.323026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6st2z\" (UniqueName: \"kubernetes.io/projected/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kube-api-access-6st2z\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.323143 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.323081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.323281 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.323147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b09e6726-a6c8-4582-aef8-02149fb5bd2e-proxy-tls\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.424374 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.424343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.424487 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.424402    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b09e6726-a6c8-4582-aef8-02149fb5bd2e-proxy-tls\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.424487 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.424433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kserve-provision-location\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.424487 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.424458    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b09e6726-a6c8-4582-aef8-02149fb5bd2e-cabundle-cert\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.424629 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.424500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6st2z\" (UniqueName: \"kubernetes.io/projected/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kube-api-access-6st2z\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.424629 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:06:38.424570    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 16 19:06:38.424719 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:06:38.424650    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b09e6726-a6c8-4582-aef8-02149fb5bd2e-proxy-tls podName:b09e6726-a6c8-4582-aef8-02149fb5bd2e nodeName:}" failed. No retries permitted until 2026-09-16 19:06:38.924631488 +0000 UTC m=+1282.738220380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b09e6726-a6c8-4582-aef8-02149fb5bd2e-proxy-tls") pod "isvc-sklearn-predictor-77674749d-kqgf4" (UID: "b09e6726-a6c8-4582-aef8-02149fb5bd2e") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 16 19:06:38.424883 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.424853    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kserve-provision-location\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.425072 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.425055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.425152 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.425135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b09e6726-a6c8-4582-aef8-02149fb5bd2e-cabundle-cert\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.435729 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.435704    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6st2z\" (UniqueName: \"kubernetes.io/projected/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kube-api-access-6st2z\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.927435 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.927412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b09e6726-a6c8-4582-aef8-02149fb5bd2e-proxy-tls\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:38.929739 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:38.929716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b09e6726-a6c8-4582-aef8-02149fb5bd2e-proxy-tls\") pod \"isvc-sklearn-predictor-77674749d-kqgf4\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:39.102861 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:39.102833    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:39.377494 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:39.377462    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"]
Sep 16 19:06:39.381308 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:06:39.381275    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09e6726_a6c8_4582_aef8_02149fb5bd2e.slice/crio-8f80718f9bb12247b1378589f3e686ad3bdd0892a8195948963525a3e5ca63a3 WatchSource:0}: Error finding container 8f80718f9bb12247b1378589f3e686ad3bdd0892a8195948963525a3e5ca63a3: Status 404 returned error can't find the container with id 8f80718f9bb12247b1378589f3e686ad3bdd0892a8195948963525a3e5ca63a3
Sep 16 19:06:39.456126 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:39.456082    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" event={"ID":"b09e6726-a6c8-4582-aef8-02149fb5bd2e","Type":"ContainerStarted","Data":"a8e05796eb34714bd5ee323e20dd91300bd1fc47174307075f3958be91f1c476"}
Sep 16 19:06:39.456227 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:39.456155    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" event={"ID":"b09e6726-a6c8-4582-aef8-02149fb5bd2e","Type":"ContainerStarted","Data":"8f80718f9bb12247b1378589f3e686ad3bdd0892a8195948963525a3e5ca63a3"}
Sep 16 19:06:43.472530 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:43.472499    2565 generic.go:356] "Generic (PLEG): container finished" podID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerID="a8e05796eb34714bd5ee323e20dd91300bd1fc47174307075f3958be91f1c476" exitCode=0
Sep 16 19:06:43.472842 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:43.472559    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" event={"ID":"b09e6726-a6c8-4582-aef8-02149fb5bd2e","Type":"ContainerDied","Data":"a8e05796eb34714bd5ee323e20dd91300bd1fc47174307075f3958be91f1c476"}
Sep 16 19:06:44.478163 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:44.478131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" event={"ID":"b09e6726-a6c8-4582-aef8-02149fb5bd2e","Type":"ContainerStarted","Data":"2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f"}
Sep 16 19:06:44.478519 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:44.478190    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" event={"ID":"b09e6726-a6c8-4582-aef8-02149fb5bd2e","Type":"ContainerStarted","Data":"1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5"}
Sep 16 19:06:44.478519 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:44.478377    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:44.498183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:44.498141    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podStartSLOduration=2.662771095 podStartE2EDuration="6.498128664s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.835357569s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:44.497875784 +0000 UTC m=+1288.311464640" watchObservedRunningTime="2026-09-16 19:06:44.498128664 +0000 UTC m=+1288.311717512"
Sep 16 19:06:45.481612 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:45.481584    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:45.482675 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:45.482649    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 19:06:46.485387 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:46.485353    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 19:06:51.489522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:51.489493    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:06:51.489990 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:06:51.489963    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 19:07:01.489941 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:07:01.489906    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 19:07:11.490479 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:07:11.490436    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 19:07:21.490821 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:07:21.490781    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 19:07:31.490916 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:07:31.490876    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 19:07:41.490760 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:07:41.490678    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 19:07:51.490230 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:07:51.490184    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 19:08:01.491307 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:01.491268    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:08:08.262156 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:08.262098    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"]
Sep 16 19:08:08.262577 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:08.262460    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container" containerID="cri-o://1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5" gracePeriod=30
Sep 16 19:08:08.262577 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:08.262547    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kube-rbac-proxy" containerID="cri-o://2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f" gracePeriod=30
Sep 16 19:08:08.786626 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:08.786592    2565 generic.go:356] "Generic (PLEG): container finished" podID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerID="2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f" exitCode=2
Sep 16 19:08:08.786764 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:08.786641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" event={"ID":"b09e6726-a6c8-4582-aef8-02149fb5bd2e","Type":"ContainerDied","Data":"2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f"}
Sep 16 19:08:11.485937 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:11.485884    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 16 19:08:11.490171 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:11.490140    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 19:08:12.194419 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.194396    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:08:12.283370 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.283344    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b09e6726-a6c8-4582-aef8-02149fb5bd2e-proxy-tls\") pod \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") "
Sep 16 19:08:12.283494 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.283390    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kserve-provision-location\") pod \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") "
Sep 16 19:08:12.283494 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.283454    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6st2z\" (UniqueName: \"kubernetes.io/projected/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kube-api-access-6st2z\") pod \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") "
Sep 16 19:08:12.283614 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.283540    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kube-rbac-proxy-sar-config\") pod \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") "
Sep 16 19:08:12.283614 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.283570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b09e6726-a6c8-4582-aef8-02149fb5bd2e-cabundle-cert\") pod \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\" (UID: \"b09e6726-a6c8-4582-aef8-02149fb5bd2e\") "
Sep 16 19:08:12.283714 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.283685    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b09e6726-a6c8-4582-aef8-02149fb5bd2e" (UID: "b09e6726-a6c8-4582-aef8-02149fb5bd2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:12.283863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.283843    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:08:12.283934 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.283884    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b09e6726-a6c8-4582-aef8-02149fb5bd2e" (UID: "b09e6726-a6c8-4582-aef8-02149fb5bd2e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:12.284026 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.284000    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09e6726-a6c8-4582-aef8-02149fb5bd2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b09e6726-a6c8-4582-aef8-02149fb5bd2e" (UID: "b09e6726-a6c8-4582-aef8-02149fb5bd2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:12.285366 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.285345    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09e6726-a6c8-4582-aef8-02149fb5bd2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b09e6726-a6c8-4582-aef8-02149fb5bd2e" (UID: "b09e6726-a6c8-4582-aef8-02149fb5bd2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:12.285449 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.285433    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kube-api-access-6st2z" (OuterVolumeSpecName: "kube-api-access-6st2z") pod "b09e6726-a6c8-4582-aef8-02149fb5bd2e" (UID: "b09e6726-a6c8-4582-aef8-02149fb5bd2e"). InnerVolumeSpecName "kube-api-access-6st2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:12.385081 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.385032    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6st2z\" (UniqueName: \"kubernetes.io/projected/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kube-api-access-6st2z\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:08:12.385081 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.385052    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b09e6726-a6c8-4582-aef8-02149fb5bd2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:08:12.385081 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.385061    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b09e6726-a6c8-4582-aef8-02149fb5bd2e-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:08:12.385081 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.385071    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b09e6726-a6c8-4582-aef8-02149fb5bd2e-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:08:12.805276 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.805238    2565 generic.go:356] "Generic (PLEG): container finished" podID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerID="1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5" exitCode=0
Sep 16 19:08:12.805603 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.805296    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"
Sep 16 19:08:12.805603 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.805335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" event={"ID":"b09e6726-a6c8-4582-aef8-02149fb5bd2e","Type":"ContainerDied","Data":"1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5"}
Sep 16 19:08:12.805603 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.805397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4" event={"ID":"b09e6726-a6c8-4582-aef8-02149fb5bd2e","Type":"ContainerDied","Data":"8f80718f9bb12247b1378589f3e686ad3bdd0892a8195948963525a3e5ca63a3"}
Sep 16 19:08:12.805603 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.805425    2565 scope.go:117] "RemoveContainer" containerID="2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f"
Sep 16 19:08:12.813401 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.813385    2565 scope.go:117] "RemoveContainer" containerID="1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5"
Sep 16 19:08:12.820667 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.820650    2565 scope.go:117] "RemoveContainer" containerID="a8e05796eb34714bd5ee323e20dd91300bd1fc47174307075f3958be91f1c476"
Sep 16 19:08:12.827451 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.827419    2565 scope.go:117] "RemoveContainer" containerID="2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f"
Sep 16 19:08:12.827944 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:08:12.827919    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f\": container with ID starting with 2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f not found: ID does not exist" containerID="2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f"
Sep 16 19:08:12.828018 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.827951    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f"} err="failed to get container status \"2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f\": rpc error: code = NotFound desc = could not find container \"2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f\": container with ID starting with 2dfc79c2808371335c4abce70762af5aa56c5ad9f9a8e37ff0537a97ffc5682f not found: ID does not exist"
Sep 16 19:08:12.828018 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.827968    2565 scope.go:117] "RemoveContainer" containerID="1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5"
Sep 16 19:08:12.828292 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:08:12.828273    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5\": container with ID starting with 1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5 not found: ID does not exist" containerID="1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5"
Sep 16 19:08:12.828392 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.828298    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5"} err="failed to get container status \"1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5\": rpc error: code = NotFound desc = could not find container \"1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5\": container with ID starting with 1f5b5df3f6252ac2799806d45e1587626b0a5f3a2bb671072e592218180329a5 not found: ID does not exist"
Sep 16 19:08:12.828392 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.828313    2565 scope.go:117] "RemoveContainer" containerID="a8e05796eb34714bd5ee323e20dd91300bd1fc47174307075f3958be91f1c476"
Sep 16 19:08:12.828590 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:08:12.828575    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e05796eb34714bd5ee323e20dd91300bd1fc47174307075f3958be91f1c476\": container with ID starting with a8e05796eb34714bd5ee323e20dd91300bd1fc47174307075f3958be91f1c476 not found: ID does not exist" containerID="a8e05796eb34714bd5ee323e20dd91300bd1fc47174307075f3958be91f1c476"
Sep 16 19:08:12.828628 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.828593    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e05796eb34714bd5ee323e20dd91300bd1fc47174307075f3958be91f1c476"} err="failed to get container status \"a8e05796eb34714bd5ee323e20dd91300bd1fc47174307075f3958be91f1c476\": rpc error: code = NotFound desc = could not find container \"a8e05796eb34714bd5ee323e20dd91300bd1fc47174307075f3958be91f1c476\": container with ID starting with a8e05796eb34714bd5ee323e20dd91300bd1fc47174307075f3958be91f1c476 not found: ID does not exist"
Sep 16 19:08:12.829459 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.829440    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"]
Sep 16 19:08:12.832553 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:12.832524    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-77674749d-kqgf4"]
Sep 16 19:08:13.424854 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.424827    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"]
Sep 16 19:08:13.425079 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.425066    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="storage-initializer"
Sep 16 19:08:13.425150 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.425080    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="storage-initializer"
Sep 16 19:08:13.425150 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.425089    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kube-rbac-proxy"
Sep 16 19:08:13.425150 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.425095    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kube-rbac-proxy"
Sep 16 19:08:13.425150 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.425131    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container"
Sep 16 19:08:13.425150 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.425138    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container"
Sep 16 19:08:13.425316 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.425197    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kserve-container"
Sep 16 19:08:13.425316 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.425206    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" containerName="kube-rbac-proxy"
Sep 16 19:08:13.429420 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.429404    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.433334 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.433306    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:08:13.433334 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.433311    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 16 19:08:13.433518 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.433353    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:08:13.433518 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.433358    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:08:13.433518 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.433371    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:08:13.435755 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.435632    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:08:13.435755 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.435689    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:08:13.439486 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.439463    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"]
Sep 16 19:08:13.494276 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.494255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679f0311-78fa-4171-bda7-bf35381ddb62-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.494368 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.494287    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcrxc\" (UniqueName: \"kubernetes.io/projected/679f0311-78fa-4171-bda7-bf35381ddb62-kube-api-access-wcrxc\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.494430 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.494377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679f0311-78fa-4171-bda7-bf35381ddb62-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.494430 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.494420    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679f0311-78fa-4171-bda7-bf35381ddb62-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.494524 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.494453    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/679f0311-78fa-4171-bda7-bf35381ddb62-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.595342 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.595303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679f0311-78fa-4171-bda7-bf35381ddb62-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.595426 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.595353    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcrxc\" (UniqueName: \"kubernetes.io/projected/679f0311-78fa-4171-bda7-bf35381ddb62-kube-api-access-wcrxc\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.595524 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.595482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679f0311-78fa-4171-bda7-bf35381ddb62-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.595524 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.595529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679f0311-78fa-4171-bda7-bf35381ddb62-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.595705 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.595554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/679f0311-78fa-4171-bda7-bf35381ddb62-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.595744 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:08:13.595718    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:08:13.595828 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:08:13.595813    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/679f0311-78fa-4171-bda7-bf35381ddb62-proxy-tls podName:679f0311-78fa-4171-bda7-bf35381ddb62 nodeName:}" failed. No retries permitted until 2026-09-16 19:08:14.095791981 +0000 UTC m=+1377.909380817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/679f0311-78fa-4171-bda7-bf35381ddb62-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" (UID: "679f0311-78fa-4171-bda7-bf35381ddb62") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:08:13.595911 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.595890    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679f0311-78fa-4171-bda7-bf35381ddb62-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.596148 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.596101    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679f0311-78fa-4171-bda7-bf35381ddb62-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.596200 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.596160    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/679f0311-78fa-4171-bda7-bf35381ddb62-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:13.604915 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:13.604890    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcrxc\" (UniqueName: \"kubernetes.io/projected/679f0311-78fa-4171-bda7-bf35381ddb62-kube-api-access-wcrxc\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:14.100508 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:14.100477    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679f0311-78fa-4171-bda7-bf35381ddb62-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:14.102824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:14.102803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679f0311-78fa-4171-bda7-bf35381ddb62-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:14.339279 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:14.339249    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:08:14.620256 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:14.620183    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"]
Sep 16 19:08:14.624788 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:08:14.624756    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679f0311_78fa_4171_bda7_bf35381ddb62.slice/crio-5939c11723637d6a84d986c75735bdd091f9994abdd283b3961f7f47457b7e97 WatchSource:0}: Error finding container 5939c11723637d6a84d986c75735bdd091f9994abdd283b3961f7f47457b7e97: Status 404 returned error can't find the container with id 5939c11723637d6a84d986c75735bdd091f9994abdd283b3961f7f47457b7e97
Sep 16 19:08:14.752028 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:14.752001    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b09e6726-a6c8-4582-aef8-02149fb5bd2e" path="/var/lib/kubelet/pods/b09e6726-a6c8-4582-aef8-02149fb5bd2e/volumes"
Sep 16 19:08:14.815214 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:14.815188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" event={"ID":"679f0311-78fa-4171-bda7-bf35381ddb62","Type":"ContainerStarted","Data":"64625817a9c23d5a184b6ba30509a13de7f4b333c61046e36ce8014310aadaa9"}
Sep 16 19:08:14.815335 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:14.815235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" event={"ID":"679f0311-78fa-4171-bda7-bf35381ddb62","Type":"ContainerStarted","Data":"5939c11723637d6a84d986c75735bdd091f9994abdd283b3961f7f47457b7e97"}
Sep 16 19:08:18.831416 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:18.831377    2565 generic.go:356] "Generic (PLEG): container finished" podID="679f0311-78fa-4171-bda7-bf35381ddb62" containerID="64625817a9c23d5a184b6ba30509a13de7f4b333c61046e36ce8014310aadaa9" exitCode=0
Sep 16 19:08:18.831788 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:18.831448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" event={"ID":"679f0311-78fa-4171-bda7-bf35381ddb62","Type":"ContainerDied","Data":"64625817a9c23d5a184b6ba30509a13de7f4b333c61046e36ce8014310aadaa9"}
Sep 16 19:08:18.832704 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:08:18.832687    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:10:15.298975 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:10:15.298936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" event={"ID":"679f0311-78fa-4171-bda7-bf35381ddb62","Type":"ContainerStarted","Data":"6bf2d50bd31ca8357641c0b97b1cfecd99b7dd3f738253acb4b0241f2ae92545"}
Sep 16 19:10:15.299516 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:10:15.299002    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" event={"ID":"679f0311-78fa-4171-bda7-bf35381ddb62","Type":"ContainerStarted","Data":"7ef556bf81d1911e4d4c3863a1ae5a9b3c60c8f7b35415f26e4512d835ee5055"}
Sep 16 19:10:15.299516 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:10:15.299080    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:10:15.299516 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:10:15.299146    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:10:15.323957 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:10:15.323913    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" podStartSLOduration=2.974553158 podStartE2EDuration="2m2.323901951s" totalImagesPullingTime="1m55.444948466s" totalInitContainerRuntime="3.904400327s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:15.322482906 +0000 UTC m=+1499.136071766" watchObservedRunningTime="2026-09-16 19:10:15.323901951 +0000 UTC m=+1499.137490807"
Sep 16 19:10:21.309117 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:10:21.309073    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:10:51.345978 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:10:51.345886    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:11:01.311910 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:01.311875    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:11:03.575164 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:03.575130    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"]
Sep 16 19:11:03.577473 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:03.575360    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" containerName="kserve-container" containerID="cri-o://7ef556bf81d1911e4d4c3863a1ae5a9b3c60c8f7b35415f26e4512d835ee5055" gracePeriod=30
Sep 16 19:11:03.577473 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:03.575394    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" containerName="kube-rbac-proxy" containerID="cri-o://6bf2d50bd31ca8357641c0b97b1cfecd99b7dd3f738253acb4b0241f2ae92545" gracePeriod=30
Sep 16 19:11:04.471988 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:04.471954    2565 generic.go:356] "Generic (PLEG): container finished" podID="679f0311-78fa-4171-bda7-bf35381ddb62" containerID="6bf2d50bd31ca8357641c0b97b1cfecd99b7dd3f738253acb4b0241f2ae92545" exitCode=2
Sep 16 19:11:04.472203 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:04.472036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" event={"ID":"679f0311-78fa-4171-bda7-bf35381ddb62","Type":"ContainerDied","Data":"6bf2d50bd31ca8357641c0b97b1cfecd99b7dd3f738253acb4b0241f2ae92545"}
Sep 16 19:11:06.303938 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:06.303897    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 16 19:11:08.732326 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.732295    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"]
Sep 16 19:11:08.735564 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.735547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.738383 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.738364    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 16 19:11:08.738488 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.738397    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:11:08.751224 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.751202    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"]
Sep 16 19:11:08.824549 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.824527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.824649 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.824556    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.824649 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.824581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td7pd\" (UniqueName: \"kubernetes.io/projected/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kube-api-access-td7pd\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.824745 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.824662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.824745 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.824688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.925041 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.925018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.925155 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.925047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.925155 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.925070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td7pd\" (UniqueName: \"kubernetes.io/projected/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kube-api-access-td7pd\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.925155 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.925094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.925155 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.925132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.925363 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:11:08.925228    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 16 19:11:08.925363 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:11:08.925305    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-proxy-tls podName:2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6 nodeName:}" failed. No retries permitted until 2026-09-16 19:11:09.425272272 +0000 UTC m=+1553.238861129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-proxy-tls") pod "isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" (UID: "2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 16 19:11:08.925498 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.925472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.925722 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.925703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.925766 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.925750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:08.940135 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:08.940094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td7pd\" (UniqueName: \"kubernetes.io/projected/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kube-api-access-td7pd\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:09.428163 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.428122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:09.430636 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.430609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:09.492226 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.492202    2565 generic.go:356] "Generic (PLEG): container finished" podID="679f0311-78fa-4171-bda7-bf35381ddb62" containerID="7ef556bf81d1911e4d4c3863a1ae5a9b3c60c8f7b35415f26e4512d835ee5055" exitCode=0
Sep 16 19:11:09.492334 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.492269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" event={"ID":"679f0311-78fa-4171-bda7-bf35381ddb62","Type":"ContainerDied","Data":"7ef556bf81d1911e4d4c3863a1ae5a9b3c60c8f7b35415f26e4512d835ee5055"}
Sep 16 19:11:09.606499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.606479    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:11:09.629308 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.629287    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679f0311-78fa-4171-bda7-bf35381ddb62-kserve-provision-location\") pod \"679f0311-78fa-4171-bda7-bf35381ddb62\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") "
Sep 16 19:11:09.629424 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.629323    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679f0311-78fa-4171-bda7-bf35381ddb62-proxy-tls\") pod \"679f0311-78fa-4171-bda7-bf35381ddb62\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") "
Sep 16 19:11:09.629424 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.629353    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcrxc\" (UniqueName: \"kubernetes.io/projected/679f0311-78fa-4171-bda7-bf35381ddb62-kube-api-access-wcrxc\") pod \"679f0311-78fa-4171-bda7-bf35381ddb62\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") "
Sep 16 19:11:09.629424 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.629377    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/679f0311-78fa-4171-bda7-bf35381ddb62-cabundle-cert\") pod \"679f0311-78fa-4171-bda7-bf35381ddb62\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") "
Sep 16 19:11:09.629584 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.629440    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679f0311-78fa-4171-bda7-bf35381ddb62-kube-rbac-proxy-sar-config\") pod \"679f0311-78fa-4171-bda7-bf35381ddb62\" (UID: \"679f0311-78fa-4171-bda7-bf35381ddb62\") "
Sep 16 19:11:09.629690 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.629653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679f0311-78fa-4171-bda7-bf35381ddb62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "679f0311-78fa-4171-bda7-bf35381ddb62" (UID: "679f0311-78fa-4171-bda7-bf35381ddb62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:09.629873 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.629841    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679f0311-78fa-4171-bda7-bf35381ddb62-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "679f0311-78fa-4171-bda7-bf35381ddb62" (UID: "679f0311-78fa-4171-bda7-bf35381ddb62"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:09.630030 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.629999    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679f0311-78fa-4171-bda7-bf35381ddb62-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "679f0311-78fa-4171-bda7-bf35381ddb62" (UID: "679f0311-78fa-4171-bda7-bf35381ddb62"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:09.631531 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.631500    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679f0311-78fa-4171-bda7-bf35381ddb62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "679f0311-78fa-4171-bda7-bf35381ddb62" (UID: "679f0311-78fa-4171-bda7-bf35381ddb62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:09.631629 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.631528    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679f0311-78fa-4171-bda7-bf35381ddb62-kube-api-access-wcrxc" (OuterVolumeSpecName: "kube-api-access-wcrxc") pod "679f0311-78fa-4171-bda7-bf35381ddb62" (UID: "679f0311-78fa-4171-bda7-bf35381ddb62"). InnerVolumeSpecName "kube-api-access-wcrxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:09.644657 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.644641    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:09.730604 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.730573    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679f0311-78fa-4171-bda7-bf35381ddb62-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:11:09.730604 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.730601    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcrxc\" (UniqueName: \"kubernetes.io/projected/679f0311-78fa-4171-bda7-bf35381ddb62-kube-api-access-wcrxc\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:11:09.730737 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.730610    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/679f0311-78fa-4171-bda7-bf35381ddb62-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:11:09.730737 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.730619    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679f0311-78fa-4171-bda7-bf35381ddb62-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:11:09.730737 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.730628    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679f0311-78fa-4171-bda7-bf35381ddb62-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:11:09.910542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:09.910520    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"]
Sep 16 19:11:09.912625 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:11:09.912598    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1cf3dd_dc7c_4b0d_8476_ae8459475aa6.slice/crio-c28967d5061574aa5327c9783aee89888b4cd970c9665d0ebaf6e6c59e1ce138 WatchSource:0}: Error finding container c28967d5061574aa5327c9783aee89888b4cd970c9665d0ebaf6e6c59e1ce138: Status 404 returned error can't find the container with id c28967d5061574aa5327c9783aee89888b4cd970c9665d0ebaf6e6c59e1ce138
Sep 16 19:11:10.498072 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:10.498038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" event={"ID":"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6","Type":"ContainerStarted","Data":"40322fa8ff26be3f28e9873b03f309d412424851639e4918010bd1ea0d3f8b4b"}
Sep 16 19:11:10.498245 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:10.498119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" event={"ID":"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6","Type":"ContainerStarted","Data":"c28967d5061574aa5327c9783aee89888b4cd970c9665d0ebaf6e6c59e1ce138"}
Sep 16 19:11:10.500118 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:10.500083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml" event={"ID":"679f0311-78fa-4171-bda7-bf35381ddb62","Type":"ContainerDied","Data":"5939c11723637d6a84d986c75735bdd091f9994abdd283b3961f7f47457b7e97"}
Sep 16 19:11:10.500230 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:10.500166    2565 scope.go:117] "RemoveContainer" containerID="6bf2d50bd31ca8357641c0b97b1cfecd99b7dd3f738253acb4b0241f2ae92545"
Sep 16 19:11:10.500230 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:10.500090    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"
Sep 16 19:11:10.507496 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:10.507475    2565 scope.go:117] "RemoveContainer" containerID="7ef556bf81d1911e4d4c3863a1ae5a9b3c60c8f7b35415f26e4512d835ee5055"
Sep 16 19:11:10.514838 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:10.514817    2565 scope.go:117] "RemoveContainer" containerID="64625817a9c23d5a184b6ba30509a13de7f4b333c61046e36ce8014310aadaa9"
Sep 16 19:11:10.531976 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:10.531955    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"]
Sep 16 19:11:10.535637 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:10.535615    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-ck8ml"]
Sep 16 19:11:10.750847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:10.750795    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" path="/var/lib/kubelet/pods/679f0311-78fa-4171-bda7-bf35381ddb62/volumes"
Sep 16 19:11:16.526227 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:16.526192    2565 generic.go:356] "Generic (PLEG): container finished" podID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerID="40322fa8ff26be3f28e9873b03f309d412424851639e4918010bd1ea0d3f8b4b" exitCode=0
Sep 16 19:11:16.526632 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:16.526237    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" event={"ID":"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6","Type":"ContainerDied","Data":"40322fa8ff26be3f28e9873b03f309d412424851639e4918010bd1ea0d3f8b4b"}
Sep 16 19:11:17.532545 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:17.532512    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" event={"ID":"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6","Type":"ContainerStarted","Data":"38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749"}
Sep 16 19:11:17.532972 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:17.532579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" event={"ID":"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6","Type":"ContainerStarted","Data":"7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22"}
Sep 16 19:11:17.532972 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:17.532859    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:17.559073 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:17.559021    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" podStartSLOduration=3.651944704 podStartE2EDuration="9.559006764s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.90706206s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:17.557333783 +0000 UTC m=+1561.370922639" watchObservedRunningTime="2026-09-16 19:11:17.559006764 +0000 UTC m=+1561.372595621"
Sep 16 19:11:18.536571 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:18.536534    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:18.537815 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:18.537790    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 19:11:19.539412 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:19.539369    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 19:11:24.543937 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:24.543910    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:24.544417 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:24.544344    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 19:11:27.291876 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.291844    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"]
Sep 16 19:11:27.292277 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.292092    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" containerName="storage-initializer"
Sep 16 19:11:27.292277 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.292115    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" containerName="storage-initializer"
Sep 16 19:11:27.292277 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.292124    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" containerName="kserve-container"
Sep 16 19:11:27.292277 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.292129    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" containerName="kserve-container"
Sep 16 19:11:27.292277 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.292135    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" containerName="kube-rbac-proxy"
Sep 16 19:11:27.292277 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.292141    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" containerName="kube-rbac-proxy"
Sep 16 19:11:27.292277 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.292189    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" containerName="kserve-container"
Sep 16 19:11:27.292277 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.292195    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="679f0311-78fa-4171-bda7-bf35381ddb62" containerName="kube-rbac-proxy"
Sep 16 19:11:27.295114 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.295091    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.297811 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.297785    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:11:27.297908 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.297862    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:11:27.306157 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.306137    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"]
Sep 16 19:11:27.349291 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.349269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-cabundle-cert\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.349374 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.349301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.349374 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.349321    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.349447 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.349405    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kserve-provision-location\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.349447 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.349436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv9wf\" (UniqueName: \"kubernetes.io/projected/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kube-api-access-fv9wf\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.450518 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.450491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-cabundle-cert\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.450633 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.450534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.450633 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.450584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.450633 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.450628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kserve-provision-location\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.450794 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.450652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv9wf\" (UniqueName: \"kubernetes.io/projected/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kube-api-access-fv9wf\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.451120 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.451084    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kserve-provision-location\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.451206 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.451179    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.451308 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.451290    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-cabundle-cert\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.453007 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.452989    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.458742 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.458712    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv9wf\" (UniqueName: \"kubernetes.io/projected/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kube-api-access-fv9wf\") pod \"isvc-canary-create-predictor-58f6457999-h88lf\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.604853 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.604798    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:27.878335 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:27.878266    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"]
Sep 16 19:11:27.881767 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:11:27.881740    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7da3b90_c10f_4e1e_8f40_b5dbf230da96.slice/crio-fc5007432bea3f38897bb2bead696f4041070cc3c87d61111f613f6b950a8f5d WatchSource:0}: Error finding container fc5007432bea3f38897bb2bead696f4041070cc3c87d61111f613f6b950a8f5d: Status 404 returned error can't find the container with id fc5007432bea3f38897bb2bead696f4041070cc3c87d61111f613f6b950a8f5d
Sep 16 19:11:28.574505 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:28.574466    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" event={"ID":"f7da3b90-c10f-4e1e-8f40-b5dbf230da96","Type":"ContainerStarted","Data":"c1b6cb2af9f310b9df0f2420cb34bacc8ee7907de3a6ee68eb424c31b2284042"}
Sep 16 19:11:28.574943 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:28.574525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" event={"ID":"f7da3b90-c10f-4e1e-8f40-b5dbf230da96","Type":"ContainerStarted","Data":"fc5007432bea3f38897bb2bead696f4041070cc3c87d61111f613f6b950a8f5d"}
Sep 16 19:11:32.598248 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:32.598211    2565 generic.go:356] "Generic (PLEG): container finished" podID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerID="c1b6cb2af9f310b9df0f2420cb34bacc8ee7907de3a6ee68eb424c31b2284042" exitCode=0
Sep 16 19:11:32.598609 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:32.598288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" event={"ID":"f7da3b90-c10f-4e1e-8f40-b5dbf230da96","Type":"ContainerDied","Data":"c1b6cb2af9f310b9df0f2420cb34bacc8ee7907de3a6ee68eb424c31b2284042"}
Sep 16 19:11:33.603689 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:33.603651    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" event={"ID":"f7da3b90-c10f-4e1e-8f40-b5dbf230da96","Type":"ContainerStarted","Data":"2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97"}
Sep 16 19:11:33.604036 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:33.603734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" event={"ID":"f7da3b90-c10f-4e1e-8f40-b5dbf230da96","Type":"ContainerStarted","Data":"53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f"}
Sep 16 19:11:33.604036 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:33.603952    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:33.624874 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:33.624825    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podStartSLOduration=2.6246573079999997 podStartE2EDuration="6.624812261s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.000154953s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:33.623294543 +0000 UTC m=+1577.436883400" watchObservedRunningTime="2026-09-16 19:11:33.624812261 +0000 UTC m=+1577.438401117"
Sep 16 19:11:34.545259 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:34.545232    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:34.607610 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:34.607580    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:34.608714 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:34.608684    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:11:35.610644 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:35.610612    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:11:40.614505 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:40.614477    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:11:40.614989 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:40.614960    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:11:40.742460 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:40.742433    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"]
Sep 16 19:11:40.742686 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:40.742662    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="kube-rbac-proxy" containerID="cri-o://38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749" gracePeriod=30
Sep 16 19:11:40.742686 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:40.742656    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="kserve-container" containerID="cri-o://7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22" gracePeriod=30
Sep 16 19:11:41.635869 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.635839    2565 generic.go:356] "Generic (PLEG): container finished" podID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerID="38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749" exitCode=2
Sep 16 19:11:41.636159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.635913    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" event={"ID":"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6","Type":"ContainerDied","Data":"38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749"}
Sep 16 19:11:41.771430 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.771411    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:41.856234 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.856181    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-cabundle-cert\") pod \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") "
Sep 16 19:11:41.856234 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.856215    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-proxy-tls\") pod \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") "
Sep 16 19:11:41.856389 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.856237    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kube-rbac-proxy-sar-config\") pod \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") "
Sep 16 19:11:41.856389 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.856272    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td7pd\" (UniqueName: \"kubernetes.io/projected/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kube-api-access-td7pd\") pod \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") "
Sep 16 19:11:41.856389 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.856327    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kserve-provision-location\") pod \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\" (UID: \"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6\") "
Sep 16 19:11:41.856619 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.856597    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" (UID: "2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:41.856666 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.856524    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" (UID: "2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:41.858158 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.858138    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" (UID: "2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:41.858211 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.858182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kube-api-access-td7pd" (OuterVolumeSpecName: "kube-api-access-td7pd") pod "2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" (UID: "2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6"). InnerVolumeSpecName "kube-api-access-td7pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:41.886538 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.886514    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" (UID: "2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:41.957294 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.957274    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:11:41.957294 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.957293    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:11:41.957404 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.957302    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td7pd\" (UniqueName: \"kubernetes.io/projected/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kube-api-access-td7pd\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:11:41.957404 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.957311    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:11:41.957404 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:41.957319    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:11:42.641219 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.641183    2565 generic.go:356] "Generic (PLEG): container finished" podID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerID="7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22" exitCode=0
Sep 16 19:11:42.641569 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.641231    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"
Sep 16 19:11:42.641569 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.641271    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" event={"ID":"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6","Type":"ContainerDied","Data":"7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22"}
Sep 16 19:11:42.641569 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.641345    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2" event={"ID":"2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6","Type":"ContainerDied","Data":"c28967d5061574aa5327c9783aee89888b4cd970c9665d0ebaf6e6c59e1ce138"}
Sep 16 19:11:42.641569 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.641370    2565 scope.go:117] "RemoveContainer" containerID="38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749"
Sep 16 19:11:42.650224 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.650207    2565 scope.go:117] "RemoveContainer" containerID="7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22"
Sep 16 19:11:42.657330 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.657315    2565 scope.go:117] "RemoveContainer" containerID="40322fa8ff26be3f28e9873b03f309d412424851639e4918010bd1ea0d3f8b4b"
Sep 16 19:11:42.664211 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.664188    2565 scope.go:117] "RemoveContainer" containerID="38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749"
Sep 16 19:11:42.664525 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:11:42.664505    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749\": container with ID starting with 38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749 not found: ID does not exist" containerID="38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749"
Sep 16 19:11:42.664589 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.664533    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749"} err="failed to get container status \"38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749\": rpc error: code = NotFound desc = could not find container \"38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749\": container with ID starting with 38261d725dc26939cbca24d28640d0fa7f9920a184cf2ba8d601d17fea53d749 not found: ID does not exist"
Sep 16 19:11:42.664589 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.664549    2565 scope.go:117] "RemoveContainer" containerID="7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22"
Sep 16 19:11:42.664836 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:11:42.664818    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22\": container with ID starting with 7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22 not found: ID does not exist" containerID="7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22"
Sep 16 19:11:42.664871 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.664842    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22"} err="failed to get container status \"7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22\": rpc error: code = NotFound desc = could not find container \"7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22\": container with ID starting with 7ae85790dfae99398c52333d8df2d9c8fbaa714a8d56cbc24fd472843d158d22 not found: ID does not exist"
Sep 16 19:11:42.664871 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.664859    2565 scope.go:117] "RemoveContainer" containerID="40322fa8ff26be3f28e9873b03f309d412424851639e4918010bd1ea0d3f8b4b"
Sep 16 19:11:42.665067 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.665048    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"]
Sep 16 19:11:42.665147 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:11:42.665078    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40322fa8ff26be3f28e9873b03f309d412424851639e4918010bd1ea0d3f8b4b\": container with ID starting with 40322fa8ff26be3f28e9873b03f309d412424851639e4918010bd1ea0d3f8b4b not found: ID does not exist" containerID="40322fa8ff26be3f28e9873b03f309d412424851639e4918010bd1ea0d3f8b4b"
Sep 16 19:11:42.665147 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.665095    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40322fa8ff26be3f28e9873b03f309d412424851639e4918010bd1ea0d3f8b4b"} err="failed to get container status \"40322fa8ff26be3f28e9873b03f309d412424851639e4918010bd1ea0d3f8b4b\": rpc error: code = NotFound desc = could not find container \"40322fa8ff26be3f28e9873b03f309d412424851639e4918010bd1ea0d3f8b4b\": container with ID starting with 40322fa8ff26be3f28e9873b03f309d412424851639e4918010bd1ea0d3f8b4b not found: ID does not exist"
Sep 16 19:11:42.670017 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.669998    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-f49f9d55d-mvkf2"]
Sep 16 19:11:42.751020 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:42.750992    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" path="/var/lib/kubelet/pods/2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6/volumes"
Sep 16 19:11:45.933799 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.933770    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"]
Sep 16 19:11:45.934159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.934007    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="kube-rbac-proxy"
Sep 16 19:11:45.934159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.934017    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="kube-rbac-proxy"
Sep 16 19:11:45.934159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.934036    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="kserve-container"
Sep 16 19:11:45.934159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.934041    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="kserve-container"
Sep 16 19:11:45.934159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.934051    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="storage-initializer"
Sep 16 19:11:45.934159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.934056    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="storage-initializer"
Sep 16 19:11:45.934159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.934095    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="kube-rbac-proxy"
Sep 16 19:11:45.934159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.934115    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1cf3dd-dc7c-4b0d-8476-ae8459475aa6" containerName="kserve-container"
Sep 16 19:11:45.938391 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.938374    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:45.941276 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.941255    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:11:45.941369 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.941257    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 16 19:11:45.948808 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.948784    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"]
Sep 16 19:11:45.984336 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.984314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:45.984433 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.984348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:45.984433 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.984383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqsb5\" (UniqueName: \"kubernetes.io/projected/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kube-api-access-gqsb5\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:45.984506 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.984440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:45.984506 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:45.984462    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:46.085458 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:46.085436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:46.085527 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:46.085463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:46.085527 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:46.085487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:46.085527 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:46.085515    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:46.085650 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:46.085550    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqsb5\" (UniqueName: \"kubernetes.io/projected/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kube-api-access-gqsb5\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:46.085650 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:11:46.085601    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 16 19:11:46.085738 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:11:46.085676    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-proxy-tls podName:0e73b8c4-2571-437b-b77c-e51b2d6eccc6 nodeName:}" failed. No retries permitted until 2026-09-16 19:11:46.585656518 +0000 UTC m=+1590.399245355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" (UID: "0e73b8c4-2571-437b-b77c-e51b2d6eccc6") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 16 19:11:46.085875 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:46.085858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:46.086124 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:46.086092    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:46.086177 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:46.086161    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:46.095447 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:46.095427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqsb5\" (UniqueName: \"kubernetes.io/projected/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kube-api-access-gqsb5\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:46.589462 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:46.589434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:46.591728 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:46.591707    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:46.847288 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:46.847233    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:47.124556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:47.122983    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"]
Sep 16 19:11:47.126182 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:11:47.126155    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e73b8c4_2571_437b_b77c_e51b2d6eccc6.slice/crio-07073d7adcbb1bd2b62f339d76e85080247b03d7feb40a708133d57fa14e0568 WatchSource:0}: Error finding container 07073d7adcbb1bd2b62f339d76e85080247b03d7feb40a708133d57fa14e0568: Status 404 returned error can't find the container with id 07073d7adcbb1bd2b62f339d76e85080247b03d7feb40a708133d57fa14e0568
Sep 16 19:11:47.662279 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:47.662243    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" event={"ID":"0e73b8c4-2571-437b-b77c-e51b2d6eccc6","Type":"ContainerStarted","Data":"ea19b411c34b7b159698199b5233b123488f2d6f3099dd6337b6ef616e429f05"}
Sep 16 19:11:47.662429 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:47.662324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" event={"ID":"0e73b8c4-2571-437b-b77c-e51b2d6eccc6","Type":"ContainerStarted","Data":"07073d7adcbb1bd2b62f339d76e85080247b03d7feb40a708133d57fa14e0568"}
Sep 16 19:11:50.615595 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:50.615549    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:11:51.677876 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:51.677841    2565 generic.go:356] "Generic (PLEG): container finished" podID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerID="ea19b411c34b7b159698199b5233b123488f2d6f3099dd6337b6ef616e429f05" exitCode=0
Sep 16 19:11:51.678294 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:51.677900    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" event={"ID":"0e73b8c4-2571-437b-b77c-e51b2d6eccc6","Type":"ContainerDied","Data":"ea19b411c34b7b159698199b5233b123488f2d6f3099dd6337b6ef616e429f05"}
Sep 16 19:11:52.683402 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:52.683367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" event={"ID":"0e73b8c4-2571-437b-b77c-e51b2d6eccc6","Type":"ContainerStarted","Data":"56b1a1f7af4a4039846a8258e0554efeae233df2e73c329d53317a0df43501a6"}
Sep 16 19:11:52.683799 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:52.683425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" event={"ID":"0e73b8c4-2571-437b-b77c-e51b2d6eccc6","Type":"ContainerStarted","Data":"9b55f137dc86ea2dda0adeed987879c15ce45ef188ca8237a4fbf3ac76e19c0c"}
Sep 16 19:11:52.683799 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:52.683619    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:52.683799 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:52.683664    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:11:52.708744 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:52.708696    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" podStartSLOduration=4.03184774 podStartE2EDuration="7.708680673s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.676832933s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:52.707884945 +0000 UTC m=+1596.521473803" watchObservedRunningTime="2026-09-16 19:11:52.708680673 +0000 UTC m=+1596.522269529"
Sep 16 19:11:58.693984 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:11:58.693946    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:12:00.615599 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:00.615546    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:12:10.615055 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:10.614948    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:12:20.615469 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:20.615429    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:12:28.746436 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:28.746371    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:12:30.615297 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:30.615248    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:12:38.697153 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:38.697100    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:12:40.615392 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:40.615348    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:12:45.747820 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:45.747780    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:12:45.977647 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:45.977621    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"]
Sep 16 19:12:45.977859 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:45.977832    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerName="kserve-container" containerID="cri-o://9b55f137dc86ea2dda0adeed987879c15ce45ef188ca8237a4fbf3ac76e19c0c" gracePeriod=30
Sep 16 19:12:45.978000 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:45.977847    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerName="kube-rbac-proxy" containerID="cri-o://56b1a1f7af4a4039846a8258e0554efeae233df2e73c329d53317a0df43501a6" gracePeriod=30
Sep 16 19:12:46.883151 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:46.883099    2565 generic.go:356] "Generic (PLEG): container finished" podID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerID="56b1a1f7af4a4039846a8258e0554efeae233df2e73c329d53317a0df43501a6" exitCode=2
Sep 16 19:12:46.883546 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:46.883173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" event={"ID":"0e73b8c4-2571-437b-b77c-e51b2d6eccc6","Type":"ContainerDied","Data":"56b1a1f7af4a4039846a8258e0554efeae233df2e73c329d53317a0df43501a6"}
Sep 16 19:12:48.688455 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:48.688409    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 16 19:12:51.207793 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.207758    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"]
Sep 16 19:12:51.210968 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.210952    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.214030 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.214011    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:12:51.214144 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.214034    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 16 19:12:51.224692 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.224671    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"]
Sep 16 19:12:51.340384 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.340358    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1712919-85bf-4c8f-8c97-6c35ad2803d7-proxy-tls\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.340502 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.340388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1712919-85bf-4c8f-8c97-6c35ad2803d7-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.340502 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.340422    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.340575 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.340505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48pvn\" (UniqueName: \"kubernetes.io/projected/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kube-api-access-48pvn\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.340575 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.340537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.441616 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.441587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.441718 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.441638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48pvn\" (UniqueName: \"kubernetes.io/projected/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kube-api-access-48pvn\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.441718 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.441657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.441718 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.441680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1712919-85bf-4c8f-8c97-6c35ad2803d7-proxy-tls\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.441718 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.441697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1712919-85bf-4c8f-8c97-6c35ad2803d7-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.441978 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.441957    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.442370 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.442350    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1712919-85bf-4c8f-8c97-6c35ad2803d7-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.442445 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.442391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.444156 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.444125    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1712919-85bf-4c8f-8c97-6c35ad2803d7-proxy-tls\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.450857 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.450833    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48pvn\" (UniqueName: \"kubernetes.io/projected/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kube-api-access-48pvn\") pod \"isvc-sklearn-v2-predictor-57c4798c78-j8cvw\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.520417 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.520354    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:51.791497 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.791458    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"]
Sep 16 19:12:51.793606 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:12:51.793578    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1712919_85bf_4c8f_8c97_6c35ad2803d7.slice/crio-63177f5f4790881f8050561a13ca67cf3e04a3776786ebd726a81e21ede8f33c WatchSource:0}: Error finding container 63177f5f4790881f8050561a13ca67cf3e04a3776786ebd726a81e21ede8f33c: Status 404 returned error can't find the container with id 63177f5f4790881f8050561a13ca67cf3e04a3776786ebd726a81e21ede8f33c
Sep 16 19:12:51.904776 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.904750    2565 generic.go:356] "Generic (PLEG): container finished" podID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerID="9b55f137dc86ea2dda0adeed987879c15ce45ef188ca8237a4fbf3ac76e19c0c" exitCode=0
Sep 16 19:12:51.904892 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.904820    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" event={"ID":"0e73b8c4-2571-437b-b77c-e51b2d6eccc6","Type":"ContainerDied","Data":"9b55f137dc86ea2dda0adeed987879c15ce45ef188ca8237a4fbf3ac76e19c0c"}
Sep 16 19:12:51.906405 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.906386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" event={"ID":"e1712919-85bf-4c8f-8c97-6c35ad2803d7","Type":"ContainerStarted","Data":"442eeb44c960952a1beb6f8a4a1cb4ce58ba3a66c1e0d03e3e66134433bcf754"}
Sep 16 19:12:51.906480 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:51.906427    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" event={"ID":"e1712919-85bf-4c8f-8c97-6c35ad2803d7","Type":"ContainerStarted","Data":"63177f5f4790881f8050561a13ca67cf3e04a3776786ebd726a81e21ede8f33c"}
Sep 16 19:12:52.002595 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.002570    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:12:52.147386 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.147322    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kube-rbac-proxy-sar-config\") pod \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") "
Sep 16 19:12:52.147527 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.147386    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqsb5\" (UniqueName: \"kubernetes.io/projected/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kube-api-access-gqsb5\") pod \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") "
Sep 16 19:12:52.147527 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.147419    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-cabundle-cert\") pod \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") "
Sep 16 19:12:52.147527 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.147439    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kserve-provision-location\") pod \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") "
Sep 16 19:12:52.147527 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.147457    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-proxy-tls\") pod \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\" (UID: \"0e73b8c4-2571-437b-b77c-e51b2d6eccc6\") "
Sep 16 19:12:52.147752 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.147726    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e73b8c4-2571-437b-b77c-e51b2d6eccc6" (UID: "0e73b8c4-2571-437b-b77c-e51b2d6eccc6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:52.147814 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.147737    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e73b8c4-2571-437b-b77c-e51b2d6eccc6" (UID: "0e73b8c4-2571-437b-b77c-e51b2d6eccc6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:52.147868 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.147852    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e73b8c4-2571-437b-b77c-e51b2d6eccc6" (UID: "0e73b8c4-2571-437b-b77c-e51b2d6eccc6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:52.149476 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.149457    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e73b8c4-2571-437b-b77c-e51b2d6eccc6" (UID: "0e73b8c4-2571-437b-b77c-e51b2d6eccc6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:52.149573 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.149549    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kube-api-access-gqsb5" (OuterVolumeSpecName: "kube-api-access-gqsb5") pod "0e73b8c4-2571-437b-b77c-e51b2d6eccc6" (UID: "0e73b8c4-2571-437b-b77c-e51b2d6eccc6"). InnerVolumeSpecName "kube-api-access-gqsb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:52.247983 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.247961    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:12:52.247983 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.247982    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:12:52.248308 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.247991    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:12:52.248308 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.248002    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:12:52.248308 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.248011    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqsb5\" (UniqueName: \"kubernetes.io/projected/0e73b8c4-2571-437b-b77c-e51b2d6eccc6-kube-api-access-gqsb5\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:12:52.912365 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.912335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k" event={"ID":"0e73b8c4-2571-437b-b77c-e51b2d6eccc6","Type":"ContainerDied","Data":"07073d7adcbb1bd2b62f339d76e85080247b03d7feb40a708133d57fa14e0568"}
Sep 16 19:12:52.912365 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.912355    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"
Sep 16 19:12:52.912525 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.912400    2565 scope.go:117] "RemoveContainer" containerID="56b1a1f7af4a4039846a8258e0554efeae233df2e73c329d53317a0df43501a6"
Sep 16 19:12:52.920643 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.920626    2565 scope.go:117] "RemoveContainer" containerID="9b55f137dc86ea2dda0adeed987879c15ce45ef188ca8237a4fbf3ac76e19c0c"
Sep 16 19:12:52.927942 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.927922    2565 scope.go:117] "RemoveContainer" containerID="ea19b411c34b7b159698199b5233b123488f2d6f3099dd6337b6ef616e429f05"
Sep 16 19:12:52.935665 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.935646    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"]
Sep 16 19:12:52.940173 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:52.940151    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6n47k"]
Sep 16 19:12:54.750350 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:54.750317    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" path="/var/lib/kubelet/pods/0e73b8c4-2571-437b-b77c-e51b2d6eccc6/volumes"
Sep 16 19:12:55.926421 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:55.926394    2565 generic.go:356] "Generic (PLEG): container finished" podID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerID="442eeb44c960952a1beb6f8a4a1cb4ce58ba3a66c1e0d03e3e66134433bcf754" exitCode=0
Sep 16 19:12:55.926727 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:55.926439    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" event={"ID":"e1712919-85bf-4c8f-8c97-6c35ad2803d7","Type":"ContainerDied","Data":"442eeb44c960952a1beb6f8a4a1cb4ce58ba3a66c1e0d03e3e66134433bcf754"}
Sep 16 19:12:56.931885 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:56.931852    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" event={"ID":"e1712919-85bf-4c8f-8c97-6c35ad2803d7","Type":"ContainerStarted","Data":"ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2"}
Sep 16 19:12:56.932253 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:56.931906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" event={"ID":"e1712919-85bf-4c8f-8c97-6c35ad2803d7","Type":"ContainerStarted","Data":"997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f"}
Sep 16 19:12:56.932253 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:56.932091    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:56.953441 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:56.953389    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podStartSLOduration=1.929390771 podStartE2EDuration="5.953371133s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.023980362s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:56.951904138 +0000 UTC m=+1660.765493023" watchObservedRunningTime="2026-09-16 19:12:56.953371133 +0000 UTC m=+1660.766959992"
Sep 16 19:12:57.935427 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:57.935397    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:12:57.936725 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:57.936698    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:12:58.938796 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:12:58.938760    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:13:03.943834 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:03.943806    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:13:03.944352 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:03.944309    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:13:12.297263 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:12.297232    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"]
Sep 16 19:13:12.297689 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:12.297470    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container" containerID="cri-o://53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f" gracePeriod=30
Sep 16 19:13:12.297689 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:12.297518    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kube-rbac-proxy" containerID="cri-o://2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97" gracePeriod=30
Sep 16 19:13:12.991240 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:12.991209    2565 generic.go:356] "Generic (PLEG): container finished" podID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerID="2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97" exitCode=2
Sep 16 19:13:12.991403 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:12.991277    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" event={"ID":"f7da3b90-c10f-4e1e-8f40-b5dbf230da96","Type":"ContainerDied","Data":"2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97"}
Sep 16 19:13:13.944766 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:13.944724    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:13:15.611319 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:15.611275    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 16 19:13:15.747356 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:15.747304    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:13:16.927639 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:16.927616    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:13:17.006730 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.006643    2565 generic.go:356] "Generic (PLEG): container finished" podID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerID="53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f" exitCode=0
Sep 16 19:13:17.006730 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.006699    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"
Sep 16 19:13:17.006937 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.006728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" event={"ID":"f7da3b90-c10f-4e1e-8f40-b5dbf230da96","Type":"ContainerDied","Data":"53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f"}
Sep 16 19:13:17.006937 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.006799    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf" event={"ID":"f7da3b90-c10f-4e1e-8f40-b5dbf230da96","Type":"ContainerDied","Data":"fc5007432bea3f38897bb2bead696f4041070cc3c87d61111f613f6b950a8f5d"}
Sep 16 19:13:17.006937 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.006832    2565 scope.go:117] "RemoveContainer" containerID="2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97"
Sep 16 19:13:17.014978 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.014959    2565 scope.go:117] "RemoveContainer" containerID="53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f"
Sep 16 19:13:17.015445 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.015426    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-proxy-tls\") pod \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") "
Sep 16 19:13:17.015582 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.015564    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kube-rbac-proxy-sar-config\") pod \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") "
Sep 16 19:13:17.015651 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.015626    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv9wf\" (UniqueName: \"kubernetes.io/projected/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kube-api-access-fv9wf\") pod \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") "
Sep 16 19:13:17.015708 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.015684    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kserve-provision-location\") pod \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") "
Sep 16 19:13:17.015765 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.015717    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-cabundle-cert\") pod \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\" (UID: \"f7da3b90-c10f-4e1e-8f40-b5dbf230da96\") "
Sep 16 19:13:17.016064 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.016036    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7da3b90-c10f-4e1e-8f40-b5dbf230da96" (UID: "f7da3b90-c10f-4e1e-8f40-b5dbf230da96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:17.016159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.016087    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f7da3b90-c10f-4e1e-8f40-b5dbf230da96" (UID: "f7da3b90-c10f-4e1e-8f40-b5dbf230da96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:17.016276 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.016154    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7da3b90-c10f-4e1e-8f40-b5dbf230da96" (UID: "f7da3b90-c10f-4e1e-8f40-b5dbf230da96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:17.017600 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.017570    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kube-api-access-fv9wf" (OuterVolumeSpecName: "kube-api-access-fv9wf") pod "f7da3b90-c10f-4e1e-8f40-b5dbf230da96" (UID: "f7da3b90-c10f-4e1e-8f40-b5dbf230da96"). InnerVolumeSpecName "kube-api-access-fv9wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:17.017687 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.017647    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7da3b90-c10f-4e1e-8f40-b5dbf230da96" (UID: "f7da3b90-c10f-4e1e-8f40-b5dbf230da96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:17.029540 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.029524    2565 scope.go:117] "RemoveContainer" containerID="c1b6cb2af9f310b9df0f2420cb34bacc8ee7907de3a6ee68eb424c31b2284042"
Sep 16 19:13:17.036439 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.036422    2565 scope.go:117] "RemoveContainer" containerID="2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97"
Sep 16 19:13:17.036738 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:13:17.036699    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97\": container with ID starting with 2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97 not found: ID does not exist" containerID="2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97"
Sep 16 19:13:17.036812 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.036750    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97"} err="failed to get container status \"2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97\": rpc error: code = NotFound desc = could not find container \"2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97\": container with ID starting with 2695470dc03ae3c0eb015bc0f3f98c58d900d8d591c2a869e2f2469bf85c7d97 not found: ID does not exist"
Sep 16 19:13:17.036812 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.036776    2565 scope.go:117] "RemoveContainer" containerID="53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f"
Sep 16 19:13:17.037069 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:13:17.037052    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f\": container with ID starting with 53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f not found: ID does not exist" containerID="53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f"
Sep 16 19:13:17.037125 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.037074    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f"} err="failed to get container status \"53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f\": rpc error: code = NotFound desc = could not find container \"53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f\": container with ID starting with 53e51475679ea9f04c12556aaf3642b2d7787d9a00fe6c5972b3ebc9e7ada03f not found: ID does not exist"
Sep 16 19:13:17.037125 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.037090    2565 scope.go:117] "RemoveContainer" containerID="c1b6cb2af9f310b9df0f2420cb34bacc8ee7907de3a6ee68eb424c31b2284042"
Sep 16 19:13:17.037290 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:13:17.037274    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b6cb2af9f310b9df0f2420cb34bacc8ee7907de3a6ee68eb424c31b2284042\": container with ID starting with c1b6cb2af9f310b9df0f2420cb34bacc8ee7907de3a6ee68eb424c31b2284042 not found: ID does not exist" containerID="c1b6cb2af9f310b9df0f2420cb34bacc8ee7907de3a6ee68eb424c31b2284042"
Sep 16 19:13:17.037330 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.037292    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b6cb2af9f310b9df0f2420cb34bacc8ee7907de3a6ee68eb424c31b2284042"} err="failed to get container status \"c1b6cb2af9f310b9df0f2420cb34bacc8ee7907de3a6ee68eb424c31b2284042\": rpc error: code = NotFound desc = could not find container \"c1b6cb2af9f310b9df0f2420cb34bacc8ee7907de3a6ee68eb424c31b2284042\": container with ID starting with c1b6cb2af9f310b9df0f2420cb34bacc8ee7907de3a6ee68eb424c31b2284042 not found: ID does not exist"
Sep 16 19:13:17.116485 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.116443    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:17.116485 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.116474    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:17.116485 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.116488    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:17.116717 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.116499    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:17.116717 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.116511    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv9wf\" (UniqueName: \"kubernetes.io/projected/f7da3b90-c10f-4e1e-8f40-b5dbf230da96-kube-api-access-fv9wf\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:13:17.329498 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.329473    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"]
Sep 16 19:13:17.331871 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.331848    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-h88lf"]
Sep 16 19:13:17.464554 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464524    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"]
Sep 16 19:13:17.464842 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464827    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container"
Sep 16 19:13:17.464918 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464846    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container"
Sep 16 19:13:17.464918 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464867    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kube-rbac-proxy"
Sep 16 19:13:17.464918 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464878    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kube-rbac-proxy"
Sep 16 19:13:17.464918 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464890    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerName="kserve-container"
Sep 16 19:13:17.464918 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464898    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerName="kserve-container"
Sep 16 19:13:17.464918 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464915    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerName="storage-initializer"
Sep 16 19:13:17.465246 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464924    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerName="storage-initializer"
Sep 16 19:13:17.465246 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464935    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="storage-initializer"
Sep 16 19:13:17.465246 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464945    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="storage-initializer"
Sep 16 19:13:17.465246 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464961    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerName="kube-rbac-proxy"
Sep 16 19:13:17.465246 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.464969    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerName="kube-rbac-proxy"
Sep 16 19:13:17.465246 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.465052    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kserve-container"
Sep 16 19:13:17.465246 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.465064    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerName="kube-rbac-proxy"
Sep 16 19:13:17.465246 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.465073    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e73b8c4-2571-437b-b77c-e51b2d6eccc6" containerName="kserve-container"
Sep 16 19:13:17.465246 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.465083    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" containerName="kube-rbac-proxy"
Sep 16 19:13:17.469498 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.469478    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.476840 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.476819    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:13:17.476919 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.476896    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:13:17.482019 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.481995    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"]
Sep 16 19:13:17.518398 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.518369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.518522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.518404    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.518522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.518430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pbdf\" (UniqueName: \"kubernetes.io/projected/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kube-api-access-5pbdf\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.518522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.518449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.518522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.518506    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.566332 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.566255    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"]
Sep 16 19:13:17.569542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.569523    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.572559 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.572542    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:13:17.583001 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.582980    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"]
Sep 16 19:13:17.619527 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.619499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9mb\" (UniqueName: \"kubernetes.io/projected/fd44feab-3fed-4054-93c5-1af3d2762a10-kube-api-access-6v9mb\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.619629 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.619538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.619629 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.619568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.619629 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.619594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.619629 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.619611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd44feab-3fed-4054-93c5-1af3d2762a10-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.619774 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.619628    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd44feab-3fed-4054-93c5-1af3d2762a10-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.619774 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.619658    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd44feab-3fed-4054-93c5-1af3d2762a10-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.619774 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.619696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pbdf\" (UniqueName: \"kubernetes.io/projected/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kube-api-access-5pbdf\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.619774 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.619729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.619774 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:13:17.619739    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 16 19:13:17.619927 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.619785    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd44feab-3fed-4054-93c5-1af3d2762a10-cabundle-cert\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.619927 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:13:17.619804    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-proxy-tls podName:f1d70c53-ab46-4fe0-b5df-ed7c70b12891 nodeName:}" failed. No retries permitted until 2026-09-16 19:13:18.119785972 +0000 UTC m=+1681.933374818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-proxy-tls") pod "isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" (UID: "f1d70c53-ab46-4fe0-b5df-ed7c70b12891") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 16 19:13:17.620052 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.620032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.620255 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.620240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.620312 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.620264    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.628457 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.628440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pbdf\" (UniqueName: \"kubernetes.io/projected/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kube-api-access-5pbdf\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:17.720332 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.720294    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9mb\" (UniqueName: \"kubernetes.io/projected/fd44feab-3fed-4054-93c5-1af3d2762a10-kube-api-access-6v9mb\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.720498 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.720354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd44feab-3fed-4054-93c5-1af3d2762a10-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.720498 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.720465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd44feab-3fed-4054-93c5-1af3d2762a10-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.720581 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.720520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd44feab-3fed-4054-93c5-1af3d2762a10-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.720623 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.720582    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd44feab-3fed-4054-93c5-1af3d2762a10-cabundle-cert\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.720713 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:13:17.720691    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:13:17.720787 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:13:17.720778    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd44feab-3fed-4054-93c5-1af3d2762a10-proxy-tls podName:fd44feab-3fed-4054-93c5-1af3d2762a10 nodeName:}" failed. No retries permitted until 2026-09-16 19:13:18.220761356 +0000 UTC m=+1682.034350212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fd44feab-3fed-4054-93c5-1af3d2762a10-proxy-tls") pod "isvc-canary-promote-predictor-5c9955746c-w4vkh" (UID: "fd44feab-3fed-4054-93c5-1af3d2762a10") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:13:17.720837 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.720800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd44feab-3fed-4054-93c5-1af3d2762a10-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.721035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.721017    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd44feab-3fed-4054-93c5-1af3d2762a10-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.721118 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.721090    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd44feab-3fed-4054-93c5-1af3d2762a10-cabundle-cert\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:17.731724 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:17.731702    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9mb\" (UniqueName: \"kubernetes.io/projected/fd44feab-3fed-4054-93c5-1af3d2762a10-kube-api-access-6v9mb\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:18.123070 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:18.123032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:18.125451 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:18.125423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:18.223752 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:18.223710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd44feab-3fed-4054-93c5-1af3d2762a10-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:18.226049 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:18.226026    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd44feab-3fed-4054-93c5-1af3d2762a10-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-w4vkh\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:18.378952 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:18.378860    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:18.478313 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:18.478251    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:18.654798 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:18.654777    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"]
Sep 16 19:13:18.656901 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:13:18.656872    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d70c53_ab46_4fe0_b5df_ed7c70b12891.slice/crio-b354795b9b18c3a064c10dae84147008d1f815746501a6bace6e9226b1bc8620 WatchSource:0}: Error finding container b354795b9b18c3a064c10dae84147008d1f815746501a6bace6e9226b1bc8620: Status 404 returned error can't find the container with id b354795b9b18c3a064c10dae84147008d1f815746501a6bace6e9226b1bc8620
Sep 16 19:13:18.752730 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:18.752696    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7da3b90-c10f-4e1e-8f40-b5dbf230da96" path="/var/lib/kubelet/pods/f7da3b90-c10f-4e1e-8f40-b5dbf230da96/volumes"
Sep 16 19:13:18.760183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:18.756893    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"]
Sep 16 19:13:19.018499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:19.018448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" event={"ID":"fd44feab-3fed-4054-93c5-1af3d2762a10","Type":"ContainerStarted","Data":"7803f8c1f733f4a8848f37eec80bcb8b6d7b65001e41ced14f4684ff289bb179"}
Sep 16 19:13:19.018712 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:19.018555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" event={"ID":"fd44feab-3fed-4054-93c5-1af3d2762a10","Type":"ContainerStarted","Data":"20354c2c3ca8767e0c2eca4ad533b085c923f123f46b439e2b790e623978607f"}
Sep 16 19:13:19.020173 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:19.020138    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" event={"ID":"f1d70c53-ab46-4fe0-b5df-ed7c70b12891","Type":"ContainerStarted","Data":"9ea1f76cf4f5dfb3311a47c4fab3521c7ccc723f2b3e24331ea37118cfe17c25"}
Sep 16 19:13:19.020329 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:19.020203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" event={"ID":"f1d70c53-ab46-4fe0-b5df-ed7c70b12891","Type":"ContainerStarted","Data":"b354795b9b18c3a064c10dae84147008d1f815746501a6bace6e9226b1bc8620"}
Sep 16 19:13:23.035833 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:23.035797    2565 generic.go:356] "Generic (PLEG): container finished" podID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerID="7803f8c1f733f4a8848f37eec80bcb8b6d7b65001e41ced14f4684ff289bb179" exitCode=0
Sep 16 19:13:23.036240 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:23.035863    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" event={"ID":"fd44feab-3fed-4054-93c5-1af3d2762a10","Type":"ContainerDied","Data":"7803f8c1f733f4a8848f37eec80bcb8b6d7b65001e41ced14f4684ff289bb179"}
Sep 16 19:13:23.037578 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:23.037560    2565 generic.go:356] "Generic (PLEG): container finished" podID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerID="9ea1f76cf4f5dfb3311a47c4fab3521c7ccc723f2b3e24331ea37118cfe17c25" exitCode=0
Sep 16 19:13:23.037683 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:23.037607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" event={"ID":"f1d70c53-ab46-4fe0-b5df-ed7c70b12891","Type":"ContainerDied","Data":"9ea1f76cf4f5dfb3311a47c4fab3521c7ccc723f2b3e24331ea37118cfe17c25"}
Sep 16 19:13:23.944929 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:23.944869    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:13:24.043723 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:24.043690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" event={"ID":"fd44feab-3fed-4054-93c5-1af3d2762a10","Type":"ContainerStarted","Data":"46d192a2ffbf7d0daf20ebc9d9786397199add43f00a74200af646b8b4d36202"}
Sep 16 19:13:24.044162 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:24.043754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" event={"ID":"fd44feab-3fed-4054-93c5-1af3d2762a10","Type":"ContainerStarted","Data":"b6ac97730ef2ecba3186d26c0026df1a210b832e515e64afa702e5dc4fc21c48"}
Sep 16 19:13:24.044162 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:24.043945    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:24.046149 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:24.046122    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" event={"ID":"f1d70c53-ab46-4fe0-b5df-ed7c70b12891","Type":"ContainerStarted","Data":"22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f"}
Sep 16 19:13:24.046281 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:24.046182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" event={"ID":"f1d70c53-ab46-4fe0-b5df-ed7c70b12891","Type":"ContainerStarted","Data":"aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56"}
Sep 16 19:13:24.046550 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:24.046518    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:24.046649 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:24.046572    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:24.048497 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:24.048457    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:13:24.066557 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:24.066517    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podStartSLOduration=3.091366143 podStartE2EDuration="7.066505352s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.975139209s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:24.06465586 +0000 UTC m=+1687.878244716" watchObservedRunningTime="2026-09-16 19:13:24.066505352 +0000 UTC m=+1687.880094207"
Sep 16 19:13:24.088454 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:24.088415    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podStartSLOduration=3.123769859 podStartE2EDuration="7.088403191s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.964633332s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:24.086690184 +0000 UTC m=+1687.900279042" watchObservedRunningTime="2026-09-16 19:13:24.088403191 +0000 UTC m=+1687.901992047"
Sep 16 19:13:25.052177 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:25.052135    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:13:25.052764 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:25.052368    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:25.053725 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:25.053702    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:13:26.056890 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:26.056854    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:13:30.057423 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:30.057396    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:13:30.057883 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:30.057857    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:13:31.060865 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:31.060840    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:13:31.061291 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:31.061265    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:13:33.945236 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:33.945197    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:13:40.057918 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:40.057833    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:13:41.061351 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:41.061304    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:13:43.944336 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:43.944274    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:13:50.058321 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:50.058281    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:13:51.061685 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:51.061647    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:13:53.944393 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:13:53.944353    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:14:00.058679 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:00.058637    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:14:01.061338 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:01.061300    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:14:03.944913 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:03.944860    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:14:10.058151 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:10.058091    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:14:11.061289 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:11.061248    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:14:13.945681 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:13.945640    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:14:20.058236 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:20.058197    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:14:21.062049 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:21.062008    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:14:21.419177 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:21.419070    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"]
Sep 16 19:14:21.419345 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:21.419324    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container" containerID="cri-o://997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f" gracePeriod=30
Sep 16 19:14:21.419392 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:21.419363    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kube-rbac-proxy" containerID="cri-o://ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2" gracePeriod=30
Sep 16 19:14:22.267777 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:22.267736    2565 generic.go:356] "Generic (PLEG): container finished" podID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerID="ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2" exitCode=2
Sep 16 19:14:22.268193 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:22.267797    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" event={"ID":"e1712919-85bf-4c8f-8c97-6c35ad2803d7","Type":"ContainerDied","Data":"ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2"}
Sep 16 19:14:23.939773 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:23.939729    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 16 19:14:23.945285 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:23.945244    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 19:14:25.861367 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:25.861341    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:14:26.010920 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.010885    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1712919-85bf-4c8f-8c97-6c35ad2803d7-cabundle-cert\") pod \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") "
Sep 16 19:14:26.011168 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.010929    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kserve-provision-location\") pod \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") "
Sep 16 19:14:26.011168 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.010987    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1712919-85bf-4c8f-8c97-6c35ad2803d7-proxy-tls\") pod \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") "
Sep 16 19:14:26.011168 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.011030    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48pvn\" (UniqueName: \"kubernetes.io/projected/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kube-api-access-48pvn\") pod \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") "
Sep 16 19:14:26.011168 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.011083    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kube-rbac-proxy-sar-config\") pod \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\" (UID: \"e1712919-85bf-4c8f-8c97-6c35ad2803d7\") "
Sep 16 19:14:26.011428 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.011391    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1712919-85bf-4c8f-8c97-6c35ad2803d7" (UID: "e1712919-85bf-4c8f-8c97-6c35ad2803d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:26.011584 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.011434    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1712919-85bf-4c8f-8c97-6c35ad2803d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e1712919-85bf-4c8f-8c97-6c35ad2803d7" (UID: "e1712919-85bf-4c8f-8c97-6c35ad2803d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:26.011584 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.011539    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e1712919-85bf-4c8f-8c97-6c35ad2803d7" (UID: "e1712919-85bf-4c8f-8c97-6c35ad2803d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:26.013122 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.013085    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1712919-85bf-4c8f-8c97-6c35ad2803d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1712919-85bf-4c8f-8c97-6c35ad2803d7" (UID: "e1712919-85bf-4c8f-8c97-6c35ad2803d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:26.013245 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.013216    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kube-api-access-48pvn" (OuterVolumeSpecName: "kube-api-access-48pvn") pod "e1712919-85bf-4c8f-8c97-6c35ad2803d7" (UID: "e1712919-85bf-4c8f-8c97-6c35ad2803d7"). InnerVolumeSpecName "kube-api-access-48pvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:26.111920 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.111888    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48pvn\" (UniqueName: \"kubernetes.io/projected/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kube-api-access-48pvn\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:14:26.111920 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.111912    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:14:26.111920 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.111922    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1712919-85bf-4c8f-8c97-6c35ad2803d7-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:14:26.112158 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.111931    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1712919-85bf-4c8f-8c97-6c35ad2803d7-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:14:26.112158 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.111941    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1712919-85bf-4c8f-8c97-6c35ad2803d7-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:14:26.284404 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.284319    2565 generic.go:356] "Generic (PLEG): container finished" podID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerID="997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f" exitCode=0
Sep 16 19:14:26.284404 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.284379    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"
Sep 16 19:14:26.284592 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.284405    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" event={"ID":"e1712919-85bf-4c8f-8c97-6c35ad2803d7","Type":"ContainerDied","Data":"997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f"}
Sep 16 19:14:26.284592 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.284461    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw" event={"ID":"e1712919-85bf-4c8f-8c97-6c35ad2803d7","Type":"ContainerDied","Data":"63177f5f4790881f8050561a13ca67cf3e04a3776786ebd726a81e21ede8f33c"}
Sep 16 19:14:26.284592 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.284488    2565 scope.go:117] "RemoveContainer" containerID="ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2"
Sep 16 19:14:26.292678 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.292664    2565 scope.go:117] "RemoveContainer" containerID="997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f"
Sep 16 19:14:26.299566 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.299549    2565 scope.go:117] "RemoveContainer" containerID="442eeb44c960952a1beb6f8a4a1cb4ce58ba3a66c1e0d03e3e66134433bcf754"
Sep 16 19:14:26.306558 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.306534    2565 scope.go:117] "RemoveContainer" containerID="ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2"
Sep 16 19:14:26.306833 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:14:26.306809    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2\": container with ID starting with ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2 not found: ID does not exist" containerID="ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2"
Sep 16 19:14:26.306907 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.306842    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2"} err="failed to get container status \"ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2\": rpc error: code = NotFound desc = could not find container \"ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2\": container with ID starting with ba84a5153f934834372889201b445069ab6535321a15ce360a06c363140d40c2 not found: ID does not exist"
Sep 16 19:14:26.306907 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.306864    2565 scope.go:117] "RemoveContainer" containerID="997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f"
Sep 16 19:14:26.307056 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.307033    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"]
Sep 16 19:14:26.307166 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:14:26.307150    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f\": container with ID starting with 997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f not found: ID does not exist" containerID="997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f"
Sep 16 19:14:26.307210 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.307173    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f"} err="failed to get container status \"997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f\": rpc error: code = NotFound desc = could not find container \"997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f\": container with ID starting with 997c1f96295a48637d0dcfe3c94ad6da4fa0b529d0e9c1c8a267b63f8204536f not found: ID does not exist"
Sep 16 19:14:26.307210 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.307187    2565 scope.go:117] "RemoveContainer" containerID="442eeb44c960952a1beb6f8a4a1cb4ce58ba3a66c1e0d03e3e66134433bcf754"
Sep 16 19:14:26.307410 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:14:26.307397    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442eeb44c960952a1beb6f8a4a1cb4ce58ba3a66c1e0d03e3e66134433bcf754\": container with ID starting with 442eeb44c960952a1beb6f8a4a1cb4ce58ba3a66c1e0d03e3e66134433bcf754 not found: ID does not exist" containerID="442eeb44c960952a1beb6f8a4a1cb4ce58ba3a66c1e0d03e3e66134433bcf754"
Sep 16 19:14:26.307448 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.307416    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442eeb44c960952a1beb6f8a4a1cb4ce58ba3a66c1e0d03e3e66134433bcf754"} err="failed to get container status \"442eeb44c960952a1beb6f8a4a1cb4ce58ba3a66c1e0d03e3e66134433bcf754\": rpc error: code = NotFound desc = could not find container \"442eeb44c960952a1beb6f8a4a1cb4ce58ba3a66c1e0d03e3e66134433bcf754\": container with ID starting with 442eeb44c960952a1beb6f8a4a1cb4ce58ba3a66c1e0d03e3e66134433bcf754 not found: ID does not exist"
Sep 16 19:14:26.310837 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.310815    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57c4798c78-j8cvw"]
Sep 16 19:14:26.751843 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:26.751812    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" path="/var/lib/kubelet/pods/e1712919-85bf-4c8f-8c97-6c35ad2803d7/volumes"
Sep 16 19:14:30.057816 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:30.057779    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:14:31.061256 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:31.061221    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:14:40.059191 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:40.059160    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:14:41.063161 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:41.063127    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:14:47.687237 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:47.687205    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"]
Sep 16 19:14:47.687718 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:47.687443    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container" containerID="cri-o://b6ac97730ef2ecba3186d26c0026df1a210b832e515e64afa702e5dc4fc21c48" gracePeriod=30
Sep 16 19:14:47.687718 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:47.687515    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kube-rbac-proxy" containerID="cri-o://46d192a2ffbf7d0daf20ebc9d9786397199add43f00a74200af646b8b4d36202" gracePeriod=30
Sep 16 19:14:48.375000 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:48.374969    2565 generic.go:356] "Generic (PLEG): container finished" podID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerID="46d192a2ffbf7d0daf20ebc9d9786397199add43f00a74200af646b8b4d36202" exitCode=2
Sep 16 19:14:48.375212 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:48.375051    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" event={"ID":"fd44feab-3fed-4054-93c5-1af3d2762a10","Type":"ContainerDied","Data":"46d192a2ffbf7d0daf20ebc9d9786397199add43f00a74200af646b8b4d36202"}
Sep 16 19:14:51.057593 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.057549    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 16 19:14:51.061802 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.061779    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:14:51.387453 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.387423    2565 generic.go:356] "Generic (PLEG): container finished" podID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerID="b6ac97730ef2ecba3186d26c0026df1a210b832e515e64afa702e5dc4fc21c48" exitCode=0
Sep 16 19:14:51.387566 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.387475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" event={"ID":"fd44feab-3fed-4054-93c5-1af3d2762a10","Type":"ContainerDied","Data":"b6ac97730ef2ecba3186d26c0026df1a210b832e515e64afa702e5dc4fc21c48"}
Sep 16 19:14:51.431526 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.431499    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:14:51.583792 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.583762    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd44feab-3fed-4054-93c5-1af3d2762a10-kube-rbac-proxy-sar-config\") pod \"fd44feab-3fed-4054-93c5-1af3d2762a10\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") "
Sep 16 19:14:51.583792 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.583795    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd44feab-3fed-4054-93c5-1af3d2762a10-kserve-provision-location\") pod \"fd44feab-3fed-4054-93c5-1af3d2762a10\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") "
Sep 16 19:14:51.583953 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.583820    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd44feab-3fed-4054-93c5-1af3d2762a10-proxy-tls\") pod \"fd44feab-3fed-4054-93c5-1af3d2762a10\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") "
Sep 16 19:14:51.583953 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.583855    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd44feab-3fed-4054-93c5-1af3d2762a10-cabundle-cert\") pod \"fd44feab-3fed-4054-93c5-1af3d2762a10\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") "
Sep 16 19:14:51.583953 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.583875    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v9mb\" (UniqueName: \"kubernetes.io/projected/fd44feab-3fed-4054-93c5-1af3d2762a10-kube-api-access-6v9mb\") pod \"fd44feab-3fed-4054-93c5-1af3d2762a10\" (UID: \"fd44feab-3fed-4054-93c5-1af3d2762a10\") "
Sep 16 19:14:51.584156 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.584128    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd44feab-3fed-4054-93c5-1af3d2762a10-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd44feab-3fed-4054-93c5-1af3d2762a10" (UID: "fd44feab-3fed-4054-93c5-1af3d2762a10"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:51.584210 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.584174    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd44feab-3fed-4054-93c5-1af3d2762a10-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd44feab-3fed-4054-93c5-1af3d2762a10" (UID: "fd44feab-3fed-4054-93c5-1af3d2762a10"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:51.584262 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.584226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd44feab-3fed-4054-93c5-1af3d2762a10-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fd44feab-3fed-4054-93c5-1af3d2762a10" (UID: "fd44feab-3fed-4054-93c5-1af3d2762a10"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:51.585917 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.585894    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd44feab-3fed-4054-93c5-1af3d2762a10-kube-api-access-6v9mb" (OuterVolumeSpecName: "kube-api-access-6v9mb") pod "fd44feab-3fed-4054-93c5-1af3d2762a10" (UID: "fd44feab-3fed-4054-93c5-1af3d2762a10"). InnerVolumeSpecName "kube-api-access-6v9mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:51.585972 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.585917    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd44feab-3fed-4054-93c5-1af3d2762a10-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd44feab-3fed-4054-93c5-1af3d2762a10" (UID: "fd44feab-3fed-4054-93c5-1af3d2762a10"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:51.684322 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.684269    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd44feab-3fed-4054-93c5-1af3d2762a10-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:14:51.684322 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.684287    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd44feab-3fed-4054-93c5-1af3d2762a10-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:14:51.684322 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.684296    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd44feab-3fed-4054-93c5-1af3d2762a10-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:14:51.684322 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.684312    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd44feab-3fed-4054-93c5-1af3d2762a10-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:14:51.684322 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:51.684321    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9mb\" (UniqueName: \"kubernetes.io/projected/fd44feab-3fed-4054-93c5-1af3d2762a10-kube-api-access-6v9mb\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:14:52.392524 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:52.392492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh" event={"ID":"fd44feab-3fed-4054-93c5-1af3d2762a10","Type":"ContainerDied","Data":"20354c2c3ca8767e0c2eca4ad533b085c923f123f46b439e2b790e623978607f"}
Sep 16 19:14:52.392868 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:52.392555    2565 scope.go:117] "RemoveContainer" containerID="46d192a2ffbf7d0daf20ebc9d9786397199add43f00a74200af646b8b4d36202"
Sep 16 19:14:52.392868 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:52.392505    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"
Sep 16 19:14:52.400980 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:52.400965    2565 scope.go:117] "RemoveContainer" containerID="b6ac97730ef2ecba3186d26c0026df1a210b832e515e64afa702e5dc4fc21c48"
Sep 16 19:14:52.407838 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:52.407822    2565 scope.go:117] "RemoveContainer" containerID="7803f8c1f733f4a8848f37eec80bcb8b6d7b65001e41ced14f4684ff289bb179"
Sep 16 19:14:52.424282 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:52.424262    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"]
Sep 16 19:14:52.428855 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:52.428836    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-w4vkh"]
Sep 16 19:14:52.751350 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:14:52.751319    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" path="/var/lib/kubelet/pods/fd44feab-3fed-4054-93c5-1af3d2762a10/volumes"
Sep 16 19:16:01.858091 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858060    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"]
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858329    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858339    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858347    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858352    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858365    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kube-rbac-proxy"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858371    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kube-rbac-proxy"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858378    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="storage-initializer"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858383    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="storage-initializer"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858391    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kube-rbac-proxy"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858397    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kube-rbac-proxy"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858406    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="storage-initializer"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858412    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="storage-initializer"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858454    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kube-rbac-proxy"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858460    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kube-rbac-proxy"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858468    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd44feab-3fed-4054-93c5-1af3d2762a10" containerName="kserve-container"
Sep 16 19:16:01.858567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.858474    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1712919-85bf-4c8f-8c97-6c35ad2803d7" containerName="kserve-container"
Sep 16 19:16:01.861429 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.861413    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:01.864159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.864139    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 16 19:16:01.864256 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.864239    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 16 19:16:01.872493 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:01.872474    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"]
Sep 16 19:16:02.016244 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.016208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e560024f-ac9f-4c3c-8a4f-38115f603ff0-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.016440 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.016255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e560024f-ac9f-4c3c-8a4f-38115f603ff0-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.016440 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.016350    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6bf5\" (UniqueName: \"kubernetes.io/projected/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kube-api-access-w6bf5\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.016440 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.016390    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.016440 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.016417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.116869 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.116776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e560024f-ac9f-4c3c-8a4f-38115f603ff0-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.116869 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.116826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6bf5\" (UniqueName: \"kubernetes.io/projected/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kube-api-access-w6bf5\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.116869 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.116864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.117172 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.116982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.117172 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.117077    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e560024f-ac9f-4c3c-8a4f-38115f603ff0-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.117401 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.117379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.117644 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.117622    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.117699 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.117622    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e560024f-ac9f-4c3c-8a4f-38115f603ff0-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.119452 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.119430    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e560024f-ac9f-4c3c-8a4f-38115f603ff0-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.131135 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.131116    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6bf5\" (UniqueName: \"kubernetes.io/projected/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kube-api-access-w6bf5\") pod \"isvc-tensorflow-predictor-547dfc5669-lccb9\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.171126 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.171074    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:02.445366 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.445310    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"]
Sep 16 19:16:02.447539 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:16:02.447509    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode560024f_ac9f_4c3c_8a4f_38115f603ff0.slice/crio-a91b6c16ed8fea5282eba593a663833aae6090db9c954636e21c05bec82bc235 WatchSource:0}: Error finding container a91b6c16ed8fea5282eba593a663833aae6090db9c954636e21c05bec82bc235: Status 404 returned error can't find the container with id a91b6c16ed8fea5282eba593a663833aae6090db9c954636e21c05bec82bc235
Sep 16 19:16:02.647614 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.647574    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" event={"ID":"e560024f-ac9f-4c3c-8a4f-38115f603ff0","Type":"ContainerStarted","Data":"26ffb17cb72bd513448d83d8c6af5e9fbc4c20978c14c3ec21045d2bc4f82d48"}
Sep 16 19:16:02.647808 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:02.647634    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" event={"ID":"e560024f-ac9f-4c3c-8a4f-38115f603ff0","Type":"ContainerStarted","Data":"a91b6c16ed8fea5282eba593a663833aae6090db9c954636e21c05bec82bc235"}
Sep 16 19:16:07.671246 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:07.671210    2565 generic.go:356] "Generic (PLEG): container finished" podID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerID="26ffb17cb72bd513448d83d8c6af5e9fbc4c20978c14c3ec21045d2bc4f82d48" exitCode=0
Sep 16 19:16:07.671725 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:07.671287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" event={"ID":"e560024f-ac9f-4c3c-8a4f-38115f603ff0","Type":"ContainerDied","Data":"26ffb17cb72bd513448d83d8c6af5e9fbc4c20978c14c3ec21045d2bc4f82d48"}
Sep 16 19:16:07.672560 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:07.672545    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:16:11.689545 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:11.689509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" event={"ID":"e560024f-ac9f-4c3c-8a4f-38115f603ff0","Type":"ContainerStarted","Data":"aafc456cc429c9b35accd10ca9491ceefeb0e86bff95d790c57e59955910b661"}
Sep 16 19:16:11.689918 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:11.689601    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" event={"ID":"e560024f-ac9f-4c3c-8a4f-38115f603ff0","Type":"ContainerStarted","Data":"63348d59bbc5a148cd86590c17ca2117500fc25caabeb20192c5d3ce86d36808"}
Sep 16 19:16:11.689918 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:11.689808    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:11.710735 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:11.710689    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podStartSLOduration=2.235372102 podStartE2EDuration="10.710674877s" totalImagesPullingTime="3.478992118s" totalInitContainerRuntime="4.996310657s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:11.709314431 +0000 UTC m=+1855.522903290" watchObservedRunningTime="2026-09-16 19:16:11.710674877 +0000 UTC m=+1855.524263736"
Sep 16 19:16:12.693630 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:12.693594    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:12.694991 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:12.694965    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:16:13.697528 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:13.697477    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:16:17.829489 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:17.829452    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"]
Sep 16 19:16:17.829953 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:17.829777    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container" containerID="cri-o://aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56" gracePeriod=30
Sep 16 19:16:17.829953 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:17.829819    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kube-rbac-proxy" containerID="cri-o://22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f" gracePeriod=30
Sep 16 19:16:18.702249 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:18.702216    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:18.702756 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:18.702718    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:16:18.719173 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:18.719140    2565 generic.go:356] "Generic (PLEG): container finished" podID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerID="22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f" exitCode=2
Sep 16 19:16:18.719309 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:18.719188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" event={"ID":"f1d70c53-ab46-4fe0-b5df-ed7c70b12891","Type":"ContainerDied","Data":"22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f"}
Sep 16 19:16:20.053047 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:20.053001    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 16 19:16:20.058428 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:20.058398    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:16:23.131551 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.131514    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"]
Sep 16 19:16:23.134713 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.134694    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.137770 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.137749    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:16:23.137873 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.137748    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 19:16:23.147310 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.147276    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"]
Sep 16 19:16:23.182159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.182124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b542829e-09f2-460a-b5f4-3d3d12c06b3d-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.182341 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.182176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b542829e-09f2-460a-b5f4-3d3d12c06b3d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.182341 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.182264    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96jlx\" (UniqueName: \"kubernetes.io/projected/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kube-api-access-96jlx\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.182341 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.182301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.182341 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.182326    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.283621 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.283576    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b542829e-09f2-460a-b5f4-3d3d12c06b3d-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.283810 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.283643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b542829e-09f2-460a-b5f4-3d3d12c06b3d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.283810 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.283699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96jlx\" (UniqueName: \"kubernetes.io/projected/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kube-api-access-96jlx\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.283810 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.283731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.283810 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:16:23.283739    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:16:23.283810 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.283765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.284081 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:16:23.283815    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b542829e-09f2-460a-b5f4-3d3d12c06b3d-proxy-tls podName:b542829e-09f2-460a-b5f4-3d3d12c06b3d nodeName:}" failed. No retries permitted until 2026-09-16 19:16:23.783792585 +0000 UTC m=+1867.597381426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b542829e-09f2-460a-b5f4-3d3d12c06b3d-proxy-tls") pod "isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" (UID: "b542829e-09f2-460a-b5f4-3d3d12c06b3d") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:16:23.284205 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.284183    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.284451 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.284428    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b542829e-09f2-460a-b5f4-3d3d12c06b3d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.284576 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.284554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.296416 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.296380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96jlx\" (UniqueName: \"kubernetes.io/projected/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kube-api-access-96jlx\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.478470 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.478438    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:16:23.485791 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.485761    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kserve-provision-location\") pod \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") "
Sep 16 19:16:23.485791 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.485800    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-proxy-tls\") pod \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") "
Sep 16 19:16:23.486033 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.485857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pbdf\" (UniqueName: \"kubernetes.io/projected/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kube-api-access-5pbdf\") pod \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") "
Sep 16 19:16:23.486033 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.485885    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kube-rbac-proxy-sar-config\") pod \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") "
Sep 16 19:16:23.486033 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.485922    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-cabundle-cert\") pod \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\" (UID: \"f1d70c53-ab46-4fe0-b5df-ed7c70b12891\") "
Sep 16 19:16:23.486219 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.486148    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1d70c53-ab46-4fe0-b5df-ed7c70b12891" (UID: "f1d70c53-ab46-4fe0-b5df-ed7c70b12891"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:23.486320 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.486297    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f1d70c53-ab46-4fe0-b5df-ed7c70b12891" (UID: "f1d70c53-ab46-4fe0-b5df-ed7c70b12891"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:23.486393 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.486364    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f1d70c53-ab46-4fe0-b5df-ed7c70b12891" (UID: "f1d70c53-ab46-4fe0-b5df-ed7c70b12891"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:23.488093 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.488068    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f1d70c53-ab46-4fe0-b5df-ed7c70b12891" (UID: "f1d70c53-ab46-4fe0-b5df-ed7c70b12891"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:23.488093 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.488070    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kube-api-access-5pbdf" (OuterVolumeSpecName: "kube-api-access-5pbdf") pod "f1d70c53-ab46-4fe0-b5df-ed7c70b12891" (UID: "f1d70c53-ab46-4fe0-b5df-ed7c70b12891"). InnerVolumeSpecName "kube-api-access-5pbdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:23.587393 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.587342    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:16:23.587393 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.587381    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:16:23.587393 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.587391    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:16:23.587393 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.587401    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pbdf\" (UniqueName: \"kubernetes.io/projected/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kube-api-access-5pbdf\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:16:23.587674 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.587412    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1d70c53-ab46-4fe0-b5df-ed7c70b12891-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:16:23.738987 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.738949    2565 generic.go:356] "Generic (PLEG): container finished" podID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerID="aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56" exitCode=0
Sep 16 19:16:23.739195 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.739014    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"
Sep 16 19:16:23.739195 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.739013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" event={"ID":"f1d70c53-ab46-4fe0-b5df-ed7c70b12891","Type":"ContainerDied","Data":"aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56"}
Sep 16 19:16:23.739195 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.739160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx" event={"ID":"f1d70c53-ab46-4fe0-b5df-ed7c70b12891","Type":"ContainerDied","Data":"b354795b9b18c3a064c10dae84147008d1f815746501a6bace6e9226b1bc8620"}
Sep 16 19:16:23.739322 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.739200    2565 scope.go:117] "RemoveContainer" containerID="22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f"
Sep 16 19:16:23.747791 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.747766    2565 scope.go:117] "RemoveContainer" containerID="aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56"
Sep 16 19:16:23.755827 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.755803    2565 scope.go:117] "RemoveContainer" containerID="9ea1f76cf4f5dfb3311a47c4fab3521c7ccc723f2b3e24331ea37118cfe17c25"
Sep 16 19:16:23.763870 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.763840    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"]
Sep 16 19:16:23.764128 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.764091    2565 scope.go:117] "RemoveContainer" containerID="22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f"
Sep 16 19:16:23.764457 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:16:23.764433    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f\": container with ID starting with 22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f not found: ID does not exist" containerID="22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f"
Sep 16 19:16:23.764546 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.764471    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f"} err="failed to get container status \"22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f\": rpc error: code = NotFound desc = could not find container \"22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f\": container with ID starting with 22b8630249994e8a5bb28acdbfd495a117a2959cfc339faa46a1ded40fbeca2f not found: ID does not exist"
Sep 16 19:16:23.764546 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.764498    2565 scope.go:117] "RemoveContainer" containerID="aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56"
Sep 16 19:16:23.764791 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:16:23.764774    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56\": container with ID starting with aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56 not found: ID does not exist" containerID="aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56"
Sep 16 19:16:23.764829 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.764799    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56"} err="failed to get container status \"aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56\": rpc error: code = NotFound desc = could not find container \"aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56\": container with ID starting with aa7ad9dc91764d60c06e0f62edbf9065d47f2da1a91f23fc5bbdc42e12a41c56 not found: ID does not exist"
Sep 16 19:16:23.764829 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.764816    2565 scope.go:117] "RemoveContainer" containerID="9ea1f76cf4f5dfb3311a47c4fab3521c7ccc723f2b3e24331ea37118cfe17c25"
Sep 16 19:16:23.765122 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:16:23.765082    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea1f76cf4f5dfb3311a47c4fab3521c7ccc723f2b3e24331ea37118cfe17c25\": container with ID starting with 9ea1f76cf4f5dfb3311a47c4fab3521c7ccc723f2b3e24331ea37118cfe17c25 not found: ID does not exist" containerID="9ea1f76cf4f5dfb3311a47c4fab3521c7ccc723f2b3e24331ea37118cfe17c25"
Sep 16 19:16:23.765194 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.765123    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea1f76cf4f5dfb3311a47c4fab3521c7ccc723f2b3e24331ea37118cfe17c25"} err="failed to get container status \"9ea1f76cf4f5dfb3311a47c4fab3521c7ccc723f2b3e24331ea37118cfe17c25\": rpc error: code = NotFound desc = could not find container \"9ea1f76cf4f5dfb3311a47c4fab3521c7ccc723f2b3e24331ea37118cfe17c25\": container with ID starting with 9ea1f76cf4f5dfb3311a47c4fab3521c7ccc723f2b3e24331ea37118cfe17c25 not found: ID does not exist"
Sep 16 19:16:23.767079 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.767059    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-cpcgx"]
Sep 16 19:16:23.789153 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.789126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b542829e-09f2-460a-b5f4-3d3d12c06b3d-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:23.791481 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:23.791458    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b542829e-09f2-460a-b5f4-3d3d12c06b3d-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-k7rj2\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:24.045098 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:24.045005    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:24.321831 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:24.321806    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"]
Sep 16 19:16:24.324384 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:16:24.324356    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb542829e_09f2_460a_b5f4_3d3d12c06b3d.slice/crio-12e621450e80df6155a6abc9dc4c5424ae9161481384538bea64670065015365 WatchSource:0}: Error finding container 12e621450e80df6155a6abc9dc4c5424ae9161481384538bea64670065015365: Status 404 returned error can't find the container with id 12e621450e80df6155a6abc9dc4c5424ae9161481384538bea64670065015365
Sep 16 19:16:24.754017 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:24.753973    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" path="/var/lib/kubelet/pods/f1d70c53-ab46-4fe0-b5df-ed7c70b12891/volumes"
Sep 16 19:16:24.754341 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:24.754323    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" event={"ID":"b542829e-09f2-460a-b5f4-3d3d12c06b3d","Type":"ContainerStarted","Data":"fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949"}
Sep 16 19:16:24.754427 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:24.754373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" event={"ID":"b542829e-09f2-460a-b5f4-3d3d12c06b3d","Type":"ContainerStarted","Data":"12e621450e80df6155a6abc9dc4c5424ae9161481384538bea64670065015365"}
Sep 16 19:16:28.703832 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:28.703797    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:28.767283 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:28.767249    2565 generic.go:356] "Generic (PLEG): container finished" podID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerID="fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949" exitCode=0
Sep 16 19:16:28.767479 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:28.767303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" event={"ID":"b542829e-09f2-460a-b5f4-3d3d12c06b3d","Type":"ContainerDied","Data":"fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949"}
Sep 16 19:16:29.773789 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:29.773753    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" event={"ID":"b542829e-09f2-460a-b5f4-3d3d12c06b3d","Type":"ContainerStarted","Data":"31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8"}
Sep 16 19:16:29.774271 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:29.773843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" event={"ID":"b542829e-09f2-460a-b5f4-3d3d12c06b3d","Type":"ContainerStarted","Data":"07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196"}
Sep 16 19:16:29.774271 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:29.774135    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:29.774271 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:29.774266    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:29.775713 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:29.775686    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:16:29.801274 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:29.801229    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podStartSLOduration=2.813456499 podStartE2EDuration="6.801214941s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.987758442s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:29.799142964 +0000 UTC m=+1873.612731822" watchObservedRunningTime="2026-09-16 19:16:29.801214941 +0000 UTC m=+1873.614803799"
Sep 16 19:16:30.777486 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:30.777446    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:16:35.787030 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:35.786997    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:16:35.787549 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:35.787499    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:16:43.688462 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:43.688377    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"]
Sep 16 19:16:43.688924 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:43.688642    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kserve-container" containerID="cri-o://63348d59bbc5a148cd86590c17ca2117500fc25caabeb20192c5d3ce86d36808" gracePeriod=30
Sep 16 19:16:43.688924 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:43.688688    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kube-rbac-proxy" containerID="cri-o://aafc456cc429c9b35accd10ca9491ceefeb0e86bff95d790c57e59955910b661" gracePeriod=30
Sep 16 19:16:43.698586 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:43.698545    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:16:43.829207 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:43.829175    2565 generic.go:356] "Generic (PLEG): container finished" podID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerID="aafc456cc429c9b35accd10ca9491ceefeb0e86bff95d790c57e59955910b661" exitCode=2
Sep 16 19:16:43.829385 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:43.829261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" event={"ID":"e560024f-ac9f-4c3c-8a4f-38115f603ff0","Type":"ContainerDied","Data":"aafc456cc429c9b35accd10ca9491ceefeb0e86bff95d790c57e59955910b661"}
Sep 16 19:16:45.788472 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:45.788433    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:16:48.698353 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:48.698307    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:16:53.698289 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:53.698237    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:16:53.698760 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:53.698383    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:16:55.787988 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:55.787942    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:16:58.698563 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:16:58.698508    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:17:03.698313 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:03.698261    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:17:05.788268 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:05.788223    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:17:08.698790 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:08.698740    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:17:13.697799 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:13.697757    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:17:13.944688 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:13.944643    2565 generic.go:356] "Generic (PLEG): container finished" podID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerID="63348d59bbc5a148cd86590c17ca2117500fc25caabeb20192c5d3ce86d36808" exitCode=137
Sep 16 19:17:13.944870 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:13.944715    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" event={"ID":"e560024f-ac9f-4c3c-8a4f-38115f603ff0","Type":"ContainerDied","Data":"63348d59bbc5a148cd86590c17ca2117500fc25caabeb20192c5d3ce86d36808"}
Sep 16 19:17:14.335121 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.335084    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:17:14.491043 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.490930    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kube-rbac-proxy-sar-config\") pod \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") "
Sep 16 19:17:14.491043 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.491016    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6bf5\" (UniqueName: \"kubernetes.io/projected/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kube-api-access-w6bf5\") pod \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") "
Sep 16 19:17:14.491274 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.491140    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e560024f-ac9f-4c3c-8a4f-38115f603ff0-proxy-tls\") pod \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") "
Sep 16 19:17:14.491274 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.491186    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e560024f-ac9f-4c3c-8a4f-38115f603ff0-cabundle-cert\") pod \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") "
Sep 16 19:17:14.491274 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.491204    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kserve-provision-location\") pod \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\" (UID: \"e560024f-ac9f-4c3c-8a4f-38115f603ff0\") "
Sep 16 19:17:14.491530 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.491404    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e560024f-ac9f-4c3c-8a4f-38115f603ff0" (UID: "e560024f-ac9f-4c3c-8a4f-38115f603ff0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:14.491642 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.491561    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e560024f-ac9f-4c3c-8a4f-38115f603ff0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e560024f-ac9f-4c3c-8a4f-38115f603ff0" (UID: "e560024f-ac9f-4c3c-8a4f-38115f603ff0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:14.493189 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.493157    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kube-api-access-w6bf5" (OuterVolumeSpecName: "kube-api-access-w6bf5") pod "e560024f-ac9f-4c3c-8a4f-38115f603ff0" (UID: "e560024f-ac9f-4c3c-8a4f-38115f603ff0"). InnerVolumeSpecName "kube-api-access-w6bf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:14.493282 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.493209    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e560024f-ac9f-4c3c-8a4f-38115f603ff0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e560024f-ac9f-4c3c-8a4f-38115f603ff0" (UID: "e560024f-ac9f-4c3c-8a4f-38115f603ff0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:14.502322 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.502299    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e560024f-ac9f-4c3c-8a4f-38115f603ff0" (UID: "e560024f-ac9f-4c3c-8a4f-38115f603ff0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:14.591824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.591790    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6bf5\" (UniqueName: \"kubernetes.io/projected/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kube-api-access-w6bf5\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:14.591824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.591823    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e560024f-ac9f-4c3c-8a4f-38115f603ff0-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:14.592002 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.591835    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e560024f-ac9f-4c3c-8a4f-38115f603ff0-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:14.592002 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.591844    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:14.592002 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.591852    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e560024f-ac9f-4c3c-8a4f-38115f603ff0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:17:14.950789 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.950751    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9" event={"ID":"e560024f-ac9f-4c3c-8a4f-38115f603ff0","Type":"ContainerDied","Data":"a91b6c16ed8fea5282eba593a663833aae6090db9c954636e21c05bec82bc235"}
Sep 16 19:17:14.950789 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.950776    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"
Sep 16 19:17:14.951345 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.950865    2565 scope.go:117] "RemoveContainer" containerID="aafc456cc429c9b35accd10ca9491ceefeb0e86bff95d790c57e59955910b661"
Sep 16 19:17:14.959149 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.959130    2565 scope.go:117] "RemoveContainer" containerID="63348d59bbc5a148cd86590c17ca2117500fc25caabeb20192c5d3ce86d36808"
Sep 16 19:17:14.966388 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.966368    2565 scope.go:117] "RemoveContainer" containerID="26ffb17cb72bd513448d83d8c6af5e9fbc4c20978c14c3ec21045d2bc4f82d48"
Sep 16 19:17:14.973314 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.973289    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"]
Sep 16 19:17:14.981600 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:14.981580    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lccb9"]
Sep 16 19:17:15.788181 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:15.788141    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:17:16.754193 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:16.754156    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" path="/var/lib/kubelet/pods/e560024f-ac9f-4c3c-8a4f-38115f603ff0/volumes"
Sep 16 19:17:25.788267 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:25.788227    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:17:35.788317 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:35.788276    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.159732    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"]
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160357    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kube-rbac-proxy"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160377    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kube-rbac-proxy"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160406    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160415    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160431    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kserve-container"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160441    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kserve-container"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160461    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kube-rbac-proxy"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160472    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kube-rbac-proxy"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160487    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="storage-initializer"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160502    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="storage-initializer"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160518    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="storage-initializer"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160526    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="storage-initializer"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160639    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kube-rbac-proxy"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160650    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e560024f-ac9f-4c3c-8a4f-38115f603ff0" containerName="kserve-container"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160659    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kube-rbac-proxy"
Sep 16 19:17:36.162443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.160675    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1d70c53-ab46-4fe0-b5df-ed7c70b12891" containerName="kserve-container"
Sep 16 19:17:36.164666 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.164639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.167452 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.167427    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 16 19:17:36.167567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.167451    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 16 19:17:36.173447 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.173420    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"]
Sep 16 19:17:36.245312 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.245286    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/093755c3-8c93-424d-b5a4-be139c1162f9-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.245468 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.245330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/093755c3-8c93-424d-b5a4-be139c1162f9-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.245468 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.245351    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/093755c3-8c93-424d-b5a4-be139c1162f9-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.245468 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.245404    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8fm\" (UniqueName: \"kubernetes.io/projected/093755c3-8c93-424d-b5a4-be139c1162f9-kube-api-access-sb8fm\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.245468 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.245450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/093755c3-8c93-424d-b5a4-be139c1162f9-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.346674 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.346640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8fm\" (UniqueName: \"kubernetes.io/projected/093755c3-8c93-424d-b5a4-be139c1162f9-kube-api-access-sb8fm\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.346866 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.346679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/093755c3-8c93-424d-b5a4-be139c1162f9-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.346866 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.346759    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/093755c3-8c93-424d-b5a4-be139c1162f9-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.346866 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.346829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/093755c3-8c93-424d-b5a4-be139c1162f9-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.346866 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.346855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/093755c3-8c93-424d-b5a4-be139c1162f9-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.347347 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.347309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/093755c3-8c93-424d-b5a4-be139c1162f9-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.347519 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.347500    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/093755c3-8c93-424d-b5a4-be139c1162f9-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.347586 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.347514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/093755c3-8c93-424d-b5a4-be139c1162f9-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.349295 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.349274    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/093755c3-8c93-424d-b5a4-be139c1162f9-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.355776 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.355757    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8fm\" (UniqueName: \"kubernetes.io/projected/093755c3-8c93-424d-b5a4-be139c1162f9-kube-api-access-sb8fm\") pod \"isvc-triton-predictor-5d7db58676-75bwg\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.474702 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.474673    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:17:36.752130 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:36.752065    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"]
Sep 16 19:17:36.752256 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:17:36.752145    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093755c3_8c93_424d_b5a4_be139c1162f9.slice/crio-74083ea80df8d5e95c67244ad9a37bd34fc6c06179832ec79a74dfb115048680 WatchSource:0}: Error finding container 74083ea80df8d5e95c67244ad9a37bd34fc6c06179832ec79a74dfb115048680: Status 404 returned error can't find the container with id 74083ea80df8d5e95c67244ad9a37bd34fc6c06179832ec79a74dfb115048680
Sep 16 19:17:37.031712 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:37.031620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" event={"ID":"093755c3-8c93-424d-b5a4-be139c1162f9","Type":"ContainerStarted","Data":"bffde491c57ea42ef696108aa0128da5b246dd3b58b7723b888fe83211731122"}
Sep 16 19:17:37.031712 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:37.031699    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" event={"ID":"093755c3-8c93-424d-b5a4-be139c1162f9","Type":"ContainerStarted","Data":"74083ea80df8d5e95c67244ad9a37bd34fc6c06179832ec79a74dfb115048680"}
Sep 16 19:17:41.048657 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:41.048567    2565 generic.go:356] "Generic (PLEG): container finished" podID="093755c3-8c93-424d-b5a4-be139c1162f9" containerID="bffde491c57ea42ef696108aa0128da5b246dd3b58b7723b888fe83211731122" exitCode=0
Sep 16 19:17:41.048657 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:41.048630    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" event={"ID":"093755c3-8c93-424d-b5a4-be139c1162f9","Type":"ContainerDied","Data":"bffde491c57ea42ef696108aa0128da5b246dd3b58b7723b888fe83211731122"}
Sep 16 19:17:45.788886 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:45.788389    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:17:48.758781 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:17:48.758748    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:19:23.316064 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:23.316027    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"]
Sep 16 19:19:23.316669 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:23.316358    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container" containerID="cri-o://07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196" gracePeriod=30
Sep 16 19:19:23.316669 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:23.316570    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kube-rbac-proxy" containerID="cri-o://31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8" gracePeriod=30
Sep 16 19:19:25.778145 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:25.778085    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:19:28.552997 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.552968    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"]
Sep 16 19:19:28.562233 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.562203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.569883 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.569635    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 19:19:28.570275 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.570237    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 19:19:28.571514 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.571493    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"]
Sep 16 19:19:28.637213 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.637160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njw99\" (UniqueName: \"kubernetes.io/projected/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kube-api-access-njw99\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.637542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.637250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.637542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.637276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75381b4a-92fc-44ff-9e09-c8d72020b2ad-cabundle-cert\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.637542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.637303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.637542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.637428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75381b4a-92fc-44ff-9e09-c8d72020b2ad-proxy-tls\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.738491 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.738450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75381b4a-92fc-44ff-9e09-c8d72020b2ad-proxy-tls\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.738791 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.738525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njw99\" (UniqueName: \"kubernetes.io/projected/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kube-api-access-njw99\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.738791 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.738620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.738791 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.738650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75381b4a-92fc-44ff-9e09-c8d72020b2ad-cabundle-cert\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.738791 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.738678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.739217 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.739194    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.739440 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.739417    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75381b4a-92fc-44ff-9e09-c8d72020b2ad-cabundle-cert\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.739519 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.739444    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.741379 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.741354    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75381b4a-92fc-44ff-9e09-c8d72020b2ad-proxy-tls\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.747726 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.747688    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:19:28.747915 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.747888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njw99\" (UniqueName: \"kubernetes.io/projected/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kube-api-access-njw99\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-k8r6k\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:28.875119 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:28.875011    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:30.778035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:30.777969    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:19:32.521092 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.521059    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:19:32.570590 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.570556    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b542829e-09f2-460a-b5f4-3d3d12c06b3d-cabundle-cert\") pod \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") "
Sep 16 19:19:32.570792 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.570646    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kserve-provision-location\") pod \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") "
Sep 16 19:19:32.570792 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.570691    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96jlx\" (UniqueName: \"kubernetes.io/projected/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kube-api-access-96jlx\") pod \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") "
Sep 16 19:19:32.570792 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.570735    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kube-rbac-proxy-sar-config\") pod \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") "
Sep 16 19:19:32.570792 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.570763    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b542829e-09f2-460a-b5f4-3d3d12c06b3d-proxy-tls\") pod \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\" (UID: \"b542829e-09f2-460a-b5f4-3d3d12c06b3d\") "
Sep 16 19:19:32.570993 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.570924    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b542829e-09f2-460a-b5f4-3d3d12c06b3d" (UID: "b542829e-09f2-460a-b5f4-3d3d12c06b3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:32.571046 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.570962    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b542829e-09f2-460a-b5f4-3d3d12c06b3d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b542829e-09f2-460a-b5f4-3d3d12c06b3d" (UID: "b542829e-09f2-460a-b5f4-3d3d12c06b3d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:32.571175 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.571149    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b542829e-09f2-460a-b5f4-3d3d12c06b3d" (UID: "b542829e-09f2-460a-b5f4-3d3d12c06b3d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:32.572755 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.572726    2565 generic.go:356] "Generic (PLEG): container finished" podID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerID="31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8" exitCode=2
Sep 16 19:19:32.572755 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.572751    2565 generic.go:356] "Generic (PLEG): container finished" podID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerID="07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196" exitCode=0
Sep 16 19:19:32.572961 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.572779    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"
Sep 16 19:19:32.572961 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.572793    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" event={"ID":"b542829e-09f2-460a-b5f4-3d3d12c06b3d","Type":"ContainerDied","Data":"31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8"}
Sep 16 19:19:32.572961 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.572861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" event={"ID":"b542829e-09f2-460a-b5f4-3d3d12c06b3d","Type":"ContainerDied","Data":"07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196"}
Sep 16 19:19:32.572961 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.572892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2" event={"ID":"b542829e-09f2-460a-b5f4-3d3d12c06b3d","Type":"ContainerDied","Data":"12e621450e80df6155a6abc9dc4c5424ae9161481384538bea64670065015365"}
Sep 16 19:19:32.572961 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.572907    2565 scope.go:117] "RemoveContainer" containerID="31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8"
Sep 16 19:19:32.573318 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.573291    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b542829e-09f2-460a-b5f4-3d3d12c06b3d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b542829e-09f2-460a-b5f4-3d3d12c06b3d" (UID: "b542829e-09f2-460a-b5f4-3d3d12c06b3d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:32.573397 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.573330    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kube-api-access-96jlx" (OuterVolumeSpecName: "kube-api-access-96jlx") pod "b542829e-09f2-460a-b5f4-3d3d12c06b3d" (UID: "b542829e-09f2-460a-b5f4-3d3d12c06b3d"). InnerVolumeSpecName "kube-api-access-96jlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:32.580417 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.580387    2565 scope.go:117] "RemoveContainer" containerID="07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196"
Sep 16 19:19:32.587623 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.587608    2565 scope.go:117] "RemoveContainer" containerID="fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949"
Sep 16 19:19:32.594511 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.594489    2565 scope.go:117] "RemoveContainer" containerID="31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8"
Sep 16 19:19:32.594783 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:19:32.594765    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8\": container with ID starting with 31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8 not found: ID does not exist" containerID="31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8"
Sep 16 19:19:32.594835 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.594793    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8"} err="failed to get container status \"31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8\": rpc error: code = NotFound desc = could not find container \"31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8\": container with ID starting with 31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8 not found: ID does not exist"
Sep 16 19:19:32.594835 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.594813    2565 scope.go:117] "RemoveContainer" containerID="07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196"
Sep 16 19:19:32.595126 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:19:32.595093    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196\": container with ID starting with 07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196 not found: ID does not exist" containerID="07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196"
Sep 16 19:19:32.595171 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.595132    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196"} err="failed to get container status \"07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196\": rpc error: code = NotFound desc = could not find container \"07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196\": container with ID starting with 07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196 not found: ID does not exist"
Sep 16 19:19:32.595171 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.595149    2565 scope.go:117] "RemoveContainer" containerID="fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949"
Sep 16 19:19:32.595381 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:19:32.595364    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949\": container with ID starting with fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949 not found: ID does not exist" containerID="fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949"
Sep 16 19:19:32.595448 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.595388    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949"} err="failed to get container status \"fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949\": rpc error: code = NotFound desc = could not find container \"fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949\": container with ID starting with fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949 not found: ID does not exist"
Sep 16 19:19:32.595448 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.595410    2565 scope.go:117] "RemoveContainer" containerID="31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8"
Sep 16 19:19:32.595661 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.595643    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8"} err="failed to get container status \"31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8\": rpc error: code = NotFound desc = could not find container \"31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8\": container with ID starting with 31d59ea0f0f257a72f56e22cae1a329046ef85b256638e14e7ed1f736d2933c8 not found: ID does not exist"
Sep 16 19:19:32.595708 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.595662    2565 scope.go:117] "RemoveContainer" containerID="07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196"
Sep 16 19:19:32.595880 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.595863    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196"} err="failed to get container status \"07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196\": rpc error: code = NotFound desc = could not find container \"07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196\": container with ID starting with 07d4456d8688eaed2c7310772435973ec2cd9b6c9049619a5120bfc729e8c196 not found: ID does not exist"
Sep 16 19:19:32.595952 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.595882    2565 scope.go:117] "RemoveContainer" containerID="fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949"
Sep 16 19:19:32.596123 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.596083    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949"} err="failed to get container status \"fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949\": rpc error: code = NotFound desc = could not find container \"fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949\": container with ID starting with fe5b34e522f6d6ecff8bf7640ae2831d8aaa2a190a92169b4cad7ebcf8ddc949 not found: ID does not exist"
Sep 16 19:19:32.671691 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.671653    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b542829e-09f2-460a-b5f4-3d3d12c06b3d-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:32.671691 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.671685    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:32.671691 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.671694    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96jlx\" (UniqueName: \"kubernetes.io/projected/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kube-api-access-96jlx\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:32.671691 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.671703    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b542829e-09f2-460a-b5f4-3d3d12c06b3d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:32.671944 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.671713    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b542829e-09f2-460a-b5f4-3d3d12c06b3d-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:32.706669 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.706586    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"]
Sep 16 19:19:32.710131 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:19:32.710089    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75381b4a_92fc_44ff_9e09_c8d72020b2ad.slice/crio-76ac9b18d272de78ab7da2e556444f811f1d24c2d0cf68c4a3d8d1790886fa00 WatchSource:0}: Error finding container 76ac9b18d272de78ab7da2e556444f811f1d24c2d0cf68c4a3d8d1790886fa00: Status 404 returned error can't find the container with id 76ac9b18d272de78ab7da2e556444f811f1d24c2d0cf68c4a3d8d1790886fa00
Sep 16 19:19:32.889281 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.889238    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"]
Sep 16 19:19:32.892949 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:32.892926    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-k7rj2"]
Sep 16 19:19:33.579269 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:33.579240    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" event={"ID":"75381b4a-92fc-44ff-9e09-c8d72020b2ad","Type":"ContainerStarted","Data":"bbc717252ac652bab963977acc4c4ff7c3e4205f9a384249a5e094a882e3f478"}
Sep 16 19:19:33.579643 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:33.579294    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" event={"ID":"75381b4a-92fc-44ff-9e09-c8d72020b2ad","Type":"ContainerStarted","Data":"76ac9b18d272de78ab7da2e556444f811f1d24c2d0cf68c4a3d8d1790886fa00"}
Sep 16 19:19:34.585021 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:34.584981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" event={"ID":"093755c3-8c93-424d-b5a4-be139c1162f9","Type":"ContainerStarted","Data":"1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0"}
Sep 16 19:19:34.585437 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:34.585056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" event={"ID":"093755c3-8c93-424d-b5a4-be139c1162f9","Type":"ContainerStarted","Data":"844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43"}
Sep 16 19:19:34.606758 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:34.606708    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" podStartSLOduration=2.190409156 podStartE2EDuration="1m58.606692912s" totalImagesPullingTime="1m52.444873455s" totalInitContainerRuntime="3.971410301s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:34.605148509 +0000 UTC m=+2058.418737368" watchObservedRunningTime="2026-09-16 19:19:34.606692912 +0000 UTC m=+2058.420281769"
Sep 16 19:19:34.751824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:34.751789    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" path="/var/lib/kubelet/pods/b542829e-09f2-460a-b5f4-3d3d12c06b3d/volumes"
Sep 16 19:19:35.589123 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:35.589082    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:19:35.589508 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:35.589161    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:19:35.590349 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:35.590324    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:19:36.594553 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:36.594505    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:19:37.598897 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:37.598863    2565 generic.go:356] "Generic (PLEG): container finished" podID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerID="bbc717252ac652bab963977acc4c4ff7c3e4205f9a384249a5e094a882e3f478" exitCode=0
Sep 16 19:19:37.599396 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:37.598942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" event={"ID":"75381b4a-92fc-44ff-9e09-c8d72020b2ad","Type":"ContainerDied","Data":"bbc717252ac652bab963977acc4c4ff7c3e4205f9a384249a5e094a882e3f478"}
Sep 16 19:19:38.605807 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:38.605773    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" event={"ID":"75381b4a-92fc-44ff-9e09-c8d72020b2ad","Type":"ContainerStarted","Data":"fdce5b62be4c8a9ae3fc9639474fe8bd351f7135a34f917867712e40b0158b37"}
Sep 16 19:19:38.606294 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:38.605835    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" event={"ID":"75381b4a-92fc-44ff-9e09-c8d72020b2ad","Type":"ContainerStarted","Data":"722dad12f2648a33c2b2310ae49a213f8efb196e3c971c8039daf012ff2a473a"}
Sep 16 19:19:38.606294 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:38.606097    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:38.606294 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:38.606276    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:38.607491 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:38.607466    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:19:38.627653 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:38.627594    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podStartSLOduration=6.49847258 podStartE2EDuration="10.627578368s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.129105788s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:38.626356822 +0000 UTC m=+2062.439945680" watchObservedRunningTime="2026-09-16 19:19:38.627578368 +0000 UTC m=+2062.441167224"
Sep 16 19:19:39.610502 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:39.610422    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:19:41.599422 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:41.599386    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:19:41.600540 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:41.600512    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:19:44.615492 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:44.615458    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:19:44.616077 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:44.616046    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:19:48.672584 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:48.672553    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"]
Sep 16 19:19:48.673132 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:48.672803    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" containerName="kserve-container" containerID="cri-o://844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43" gracePeriod=30
Sep 16 19:19:48.673132 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:48.672847    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" containerName="kube-rbac-proxy" containerID="cri-o://1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0" gracePeriod=30
Sep 16 19:19:49.650658 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:49.650620    2565 generic.go:356] "Generic (PLEG): container finished" podID="093755c3-8c93-424d-b5a4-be139c1162f9" containerID="1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0" exitCode=2
Sep 16 19:19:49.650867 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:49.650689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" event={"ID":"093755c3-8c93-424d-b5a4-be139c1162f9","Type":"ContainerDied","Data":"1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0"}
Sep 16 19:19:51.361340 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.361307    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:19:51.531677 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.531647    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb8fm\" (UniqueName: \"kubernetes.io/projected/093755c3-8c93-424d-b5a4-be139c1162f9-kube-api-access-sb8fm\") pod \"093755c3-8c93-424d-b5a4-be139c1162f9\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") "
Sep 16 19:19:51.531887 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.531695    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/093755c3-8c93-424d-b5a4-be139c1162f9-proxy-tls\") pod \"093755c3-8c93-424d-b5a4-be139c1162f9\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") "
Sep 16 19:19:51.531887 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.531725    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/093755c3-8c93-424d-b5a4-be139c1162f9-cabundle-cert\") pod \"093755c3-8c93-424d-b5a4-be139c1162f9\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") "
Sep 16 19:19:51.531887 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.531776    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/093755c3-8c93-424d-b5a4-be139c1162f9-kube-rbac-proxy-sar-config\") pod \"093755c3-8c93-424d-b5a4-be139c1162f9\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") "
Sep 16 19:19:51.531887 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.531805    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/093755c3-8c93-424d-b5a4-be139c1162f9-kserve-provision-location\") pod \"093755c3-8c93-424d-b5a4-be139c1162f9\" (UID: \"093755c3-8c93-424d-b5a4-be139c1162f9\") "
Sep 16 19:19:51.532201 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.532176    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093755c3-8c93-424d-b5a4-be139c1162f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "093755c3-8c93-424d-b5a4-be139c1162f9" (UID: "093755c3-8c93-424d-b5a4-be139c1162f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:51.532201 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.532186    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093755c3-8c93-424d-b5a4-be139c1162f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "093755c3-8c93-424d-b5a4-be139c1162f9" (UID: "093755c3-8c93-424d-b5a4-be139c1162f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:51.532303 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.532286    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093755c3-8c93-424d-b5a4-be139c1162f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "093755c3-8c93-424d-b5a4-be139c1162f9" (UID: "093755c3-8c93-424d-b5a4-be139c1162f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:51.533845 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.533821    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093755c3-8c93-424d-b5a4-be139c1162f9-kube-api-access-sb8fm" (OuterVolumeSpecName: "kube-api-access-sb8fm") pod "093755c3-8c93-424d-b5a4-be139c1162f9" (UID: "093755c3-8c93-424d-b5a4-be139c1162f9"). InnerVolumeSpecName "kube-api-access-sb8fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:51.533934 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.533839    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093755c3-8c93-424d-b5a4-be139c1162f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "093755c3-8c93-424d-b5a4-be139c1162f9" (UID: "093755c3-8c93-424d-b5a4-be139c1162f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:51.632530 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.632491    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/093755c3-8c93-424d-b5a4-be139c1162f9-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:51.632530 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.632521    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb8fm\" (UniqueName: \"kubernetes.io/projected/093755c3-8c93-424d-b5a4-be139c1162f9-kube-api-access-sb8fm\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:51.632530 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.632531    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/093755c3-8c93-424d-b5a4-be139c1162f9-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:51.632530 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.632542    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/093755c3-8c93-424d-b5a4-be139c1162f9-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:51.632795 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.632551    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/093755c3-8c93-424d-b5a4-be139c1162f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:19:51.660617 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.660581    2565 generic.go:356] "Generic (PLEG): container finished" podID="093755c3-8c93-424d-b5a4-be139c1162f9" containerID="844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43" exitCode=0
Sep 16 19:19:51.660758 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.660632    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" event={"ID":"093755c3-8c93-424d-b5a4-be139c1162f9","Type":"ContainerDied","Data":"844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43"}
Sep 16 19:19:51.660758 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.660708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg" event={"ID":"093755c3-8c93-424d-b5a4-be139c1162f9","Type":"ContainerDied","Data":"74083ea80df8d5e95c67244ad9a37bd34fc6c06179832ec79a74dfb115048680"}
Sep 16 19:19:51.660758 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.660648    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"
Sep 16 19:19:51.660758 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.660737    2565 scope.go:117] "RemoveContainer" containerID="1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0"
Sep 16 19:19:51.669132 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.669097    2565 scope.go:117] "RemoveContainer" containerID="844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43"
Sep 16 19:19:51.677064 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.677043    2565 scope.go:117] "RemoveContainer" containerID="bffde491c57ea42ef696108aa0128da5b246dd3b58b7723b888fe83211731122"
Sep 16 19:19:51.684064 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.684044    2565 scope.go:117] "RemoveContainer" containerID="1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0"
Sep 16 19:19:51.684444 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:19:51.684419    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0\": container with ID starting with 1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0 not found: ID does not exist" containerID="1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0"
Sep 16 19:19:51.684549 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.684455    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0"} err="failed to get container status \"1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0\": rpc error: code = NotFound desc = could not find container \"1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0\": container with ID starting with 1fef04b91f28477df19d62225af559761aa029875989dd6ae362c86f881bb7d0 not found: ID does not exist"
Sep 16 19:19:51.684549 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.684478    2565 scope.go:117] "RemoveContainer" containerID="844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43"
Sep 16 19:19:51.684880 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:19:51.684766    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43\": container with ID starting with 844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43 not found: ID does not exist" containerID="844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43"
Sep 16 19:19:51.684880 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.684802    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43"} err="failed to get container status \"844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43\": rpc error: code = NotFound desc = could not find container \"844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43\": container with ID starting with 844e32f8d1660ae6c771e522ae75932974c6d4ce38ac63d0892d1adc18a36d43 not found: ID does not exist"
Sep 16 19:19:51.684880 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.684825    2565 scope.go:117] "RemoveContainer" containerID="bffde491c57ea42ef696108aa0128da5b246dd3b58b7723b888fe83211731122"
Sep 16 19:19:51.685167 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:19:51.685131    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bffde491c57ea42ef696108aa0128da5b246dd3b58b7723b888fe83211731122\": container with ID starting with bffde491c57ea42ef696108aa0128da5b246dd3b58b7723b888fe83211731122 not found: ID does not exist" containerID="bffde491c57ea42ef696108aa0128da5b246dd3b58b7723b888fe83211731122"
Sep 16 19:19:51.685167 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.685158    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bffde491c57ea42ef696108aa0128da5b246dd3b58b7723b888fe83211731122"} err="failed to get container status \"bffde491c57ea42ef696108aa0128da5b246dd3b58b7723b888fe83211731122\": rpc error: code = NotFound desc = could not find container \"bffde491c57ea42ef696108aa0128da5b246dd3b58b7723b888fe83211731122\": container with ID starting with bffde491c57ea42ef696108aa0128da5b246dd3b58b7723b888fe83211731122 not found: ID does not exist"
Sep 16 19:19:51.686303 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.686284    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"]
Sep 16 19:19:51.689698 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:51.689675    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-75bwg"]
Sep 16 19:19:52.752373 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:52.752340    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" path="/var/lib/kubelet/pods/093755c3-8c93-424d-b5a4-be139c1162f9/volumes"
Sep 16 19:19:53.860640 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860603    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"]
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860864    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" containerName="kserve-container"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860876    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" containerName="kserve-container"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860889    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860894    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860901    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" containerName="storage-initializer"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860906    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" containerName="storage-initializer"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860913    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" containerName="kube-rbac-proxy"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860918    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" containerName="kube-rbac-proxy"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860926    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kube-rbac-proxy"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860931    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kube-rbac-proxy"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860938    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="storage-initializer"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860943    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="storage-initializer"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860988    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kube-rbac-proxy"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.860998    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" containerName="kube-rbac-proxy"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.861004    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b542829e-09f2-460a-b5f4-3d3d12c06b3d" containerName="kserve-container"
Sep 16 19:19:53.861035 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.861010    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="093755c3-8c93-424d-b5a4-be139c1162f9" containerName="kserve-container"
Sep 16 19:19:53.863821 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.863804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:53.866373 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.866348    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 16 19:19:53.866894 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.866877    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 16 19:19:53.878128 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.878095    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"]
Sep 16 19:19:53.949477 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.949443    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:53.949477 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.949481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:53.949725 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.949503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:53.949725 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.949553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:53.949725 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:53.949625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvq8\" (UniqueName: \"kubernetes.io/projected/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kube-api-access-mxvq8\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:54.050531 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.050488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvq8\" (UniqueName: \"kubernetes.io/projected/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kube-api-access-mxvq8\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:54.050824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.050577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:54.050824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.050607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:54.050824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.050636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:54.050824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.050662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:54.051166 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.051144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:54.051426 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.051400    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:54.051538 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.051518    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:54.053518 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.053491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:54.059242 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.059219    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvq8\" (UniqueName: \"kubernetes.io/projected/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kube-api-access-mxvq8\") pod \"isvc-xgboost-predictor-7566d65bf7-4hfwk\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:54.173976 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.173889    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:19:54.447496 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.447471    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"]
Sep 16 19:19:54.449856 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:19:54.449828    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1e16e1_229b_4334_94bd_8fc1a59f40f3.slice/crio-8a9648d2c4062bf05fe66a3f739cd1a6928150f971e75b443fd05019dc324a50 WatchSource:0}: Error finding container 8a9648d2c4062bf05fe66a3f739cd1a6928150f971e75b443fd05019dc324a50: Status 404 returned error can't find the container with id 8a9648d2c4062bf05fe66a3f739cd1a6928150f971e75b443fd05019dc324a50
Sep 16 19:19:54.616707 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.616667    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:19:54.673661 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.673622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" event={"ID":"dd1e16e1-229b-4334-94bd-8fc1a59f40f3","Type":"ContainerStarted","Data":"6c2ed34a8e0e6a8e9ecbebf27204cdebd0a134566958d630974289f05d9570d7"}
Sep 16 19:19:54.673824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:54.673682    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" event={"ID":"dd1e16e1-229b-4334-94bd-8fc1a59f40f3","Type":"ContainerStarted","Data":"8a9648d2c4062bf05fe66a3f739cd1a6928150f971e75b443fd05019dc324a50"}
Sep 16 19:19:58.689306 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:58.689266    2565 generic.go:356] "Generic (PLEG): container finished" podID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerID="6c2ed34a8e0e6a8e9ecbebf27204cdebd0a134566958d630974289f05d9570d7" exitCode=0
Sep 16 19:19:58.689737 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:19:58.689340    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" event={"ID":"dd1e16e1-229b-4334-94bd-8fc1a59f40f3","Type":"ContainerDied","Data":"6c2ed34a8e0e6a8e9ecbebf27204cdebd0a134566958d630974289f05d9570d7"}
Sep 16 19:20:04.616889 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:04.616848    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:20:14.616623 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:14.616581    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:20:16.774241 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:16.774211    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" event={"ID":"dd1e16e1-229b-4334-94bd-8fc1a59f40f3","Type":"ContainerStarted","Data":"682fbe77fbc9fa7096e904b058f23ba477ce5fa5403e96d9b8fe5c2f297e6421"}
Sep 16 19:20:17.780270 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:17.780233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" event={"ID":"dd1e16e1-229b-4334-94bd-8fc1a59f40f3","Type":"ContainerStarted","Data":"87d642c5e9fb106b15203fb5c1d5d1e49377c6ec0ba10bd04f24c09af16ca198"}
Sep 16 19:20:17.780660 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:17.780395    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:20:17.801540 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:17.801482    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podStartSLOduration=2.940438968 podStartE2EDuration="24.801467029s" totalImagesPullingTime="17.905126633s" totalInitContainerRuntime="3.955901428s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:17.799651481 +0000 UTC m=+2101.613240340" watchObservedRunningTime="2026-09-16 19:20:17.801467029 +0000 UTC m=+2101.615055889"
Sep 16 19:20:18.783650 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:18.783620    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:20:18.784796 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:18.784768    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:20:19.787704 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:19.787663    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:20:24.616722 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:24.616684    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:20:24.791762 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:24.791735    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:20:24.792249 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:24.792216    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:20:34.616600 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:34.616557    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:20:34.792499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:34.792447    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:20:44.616293 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:44.616252    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:20:44.792930 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:44.792888    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:20:51.724892 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:51.724852    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"]
Sep 16 19:20:51.725442 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:51.725136    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container" containerID="cri-o://722dad12f2648a33c2b2310ae49a213f8efb196e3c971c8039daf012ff2a473a" gracePeriod=30
Sep 16 19:20:51.725442 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:51.725149    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kube-rbac-proxy" containerID="cri-o://fdce5b62be4c8a9ae3fc9639474fe8bd351f7135a34f917867712e40b0158b37" gracePeriod=30
Sep 16 19:20:51.725442 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:51.725318    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:20:51.905583 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:51.905553    2565 generic.go:356] "Generic (PLEG): container finished" podID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerID="fdce5b62be4c8a9ae3fc9639474fe8bd351f7135a34f917867712e40b0158b37" exitCode=2
Sep 16 19:20:51.905583 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:51.905595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" event={"ID":"75381b4a-92fc-44ff-9e09-c8d72020b2ad","Type":"ContainerDied","Data":"fdce5b62be4c8a9ae3fc9639474fe8bd351f7135a34f917867712e40b0158b37"}
Sep 16 19:20:54.611550 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:54.611503    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 16 19:20:54.792315 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:54.792274    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:20:59.610929 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:20:59.610875    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 16 19:21:01.727263 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:01.727228    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:21:04.611152 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:04.611073    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 16 19:21:04.611567 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:04.611240    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:21:04.792265 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:04.792229    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:21:09.610911 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:09.610806    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 16 19:21:14.610814 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:14.610760    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 16 19:21:14.792910 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:14.792864    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:21:19.610796 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:19.610740    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 16 19:21:21.877944 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:21:21.877900    2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75381b4a_92fc_44ff_9e09_c8d72020b2ad.slice/crio-conmon-722dad12f2648a33c2b2310ae49a213f8efb196e3c971c8039daf012ff2a473a.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:21:22.021792 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.021755    2565 generic.go:356] "Generic (PLEG): container finished" podID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerID="722dad12f2648a33c2b2310ae49a213f8efb196e3c971c8039daf012ff2a473a" exitCode=137
Sep 16 19:21:22.021932 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.021806    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" event={"ID":"75381b4a-92fc-44ff-9e09-c8d72020b2ad","Type":"ContainerDied","Data":"722dad12f2648a33c2b2310ae49a213f8efb196e3c971c8039daf012ff2a473a"}
Sep 16 19:21:22.035030 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.034999    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:21:22.050957 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.050920    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75381b4a-92fc-44ff-9e09-c8d72020b2ad-cabundle-cert\") pod \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") "
Sep 16 19:21:22.050957 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.050960    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njw99\" (UniqueName: \"kubernetes.io/projected/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kube-api-access-njw99\") pod \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") "
Sep 16 19:21:22.051256 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.051010    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75381b4a-92fc-44ff-9e09-c8d72020b2ad-proxy-tls\") pod \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") "
Sep 16 19:21:22.051256 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.051044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kserve-provision-location\") pod \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") "
Sep 16 19:21:22.051256 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.051126    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kube-rbac-proxy-sar-config\") pod \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\" (UID: \"75381b4a-92fc-44ff-9e09-c8d72020b2ad\") "
Sep 16 19:21:22.051476 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.051448    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75381b4a-92fc-44ff-9e09-c8d72020b2ad" (UID: "75381b4a-92fc-44ff-9e09-c8d72020b2ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:22.051476 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.051466    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75381b4a-92fc-44ff-9e09-c8d72020b2ad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75381b4a-92fc-44ff-9e09-c8d72020b2ad" (UID: "75381b4a-92fc-44ff-9e09-c8d72020b2ad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:22.051589 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.051558    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "75381b4a-92fc-44ff-9e09-c8d72020b2ad" (UID: "75381b4a-92fc-44ff-9e09-c8d72020b2ad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:22.053227 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.053204    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75381b4a-92fc-44ff-9e09-c8d72020b2ad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75381b4a-92fc-44ff-9e09-c8d72020b2ad" (UID: "75381b4a-92fc-44ff-9e09-c8d72020b2ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:22.053376 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.053359    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kube-api-access-njw99" (OuterVolumeSpecName: "kube-api-access-njw99") pod "75381b4a-92fc-44ff-9e09-c8d72020b2ad" (UID: "75381b4a-92fc-44ff-9e09-c8d72020b2ad"). InnerVolumeSpecName "kube-api-access-njw99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:22.152187 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.152149    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75381b4a-92fc-44ff-9e09-c8d72020b2ad-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:22.152187 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.152180    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njw99\" (UniqueName: \"kubernetes.io/projected/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kube-api-access-njw99\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:22.152187 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.152192    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75381b4a-92fc-44ff-9e09-c8d72020b2ad-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:22.152432 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.152203    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:22.152432 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:22.152211    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75381b4a-92fc-44ff-9e09-c8d72020b2ad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:23.028785 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:23.028696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k" event={"ID":"75381b4a-92fc-44ff-9e09-c8d72020b2ad","Type":"ContainerDied","Data":"76ac9b18d272de78ab7da2e556444f811f1d24c2d0cf68c4a3d8d1790886fa00"}
Sep 16 19:21:23.028785 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:23.028768    2565 scope.go:117] "RemoveContainer" containerID="fdce5b62be4c8a9ae3fc9639474fe8bd351f7135a34f917867712e40b0158b37"
Sep 16 19:21:23.028785 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:23.028779    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"
Sep 16 19:21:23.037346 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:23.037326    2565 scope.go:117] "RemoveContainer" containerID="722dad12f2648a33c2b2310ae49a213f8efb196e3c971c8039daf012ff2a473a"
Sep 16 19:21:23.046517 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:23.046495    2565 scope.go:117] "RemoveContainer" containerID="bbc717252ac652bab963977acc4c4ff7c3e4205f9a384249a5e094a882e3f478"
Sep 16 19:21:23.055642 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:23.055620    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"]
Sep 16 19:21:23.058288 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:23.058260    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-k8r6k"]
Sep 16 19:21:24.751592 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:24.751553    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" path="/var/lib/kubelet/pods/75381b4a-92fc-44ff-9e09-c8d72020b2ad/volumes"
Sep 16 19:21:24.793160 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:24.793133    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:21:34.015656 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:34.015605    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"]
Sep 16 19:21:34.016099 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:34.015873    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container" containerID="cri-o://682fbe77fbc9fa7096e904b058f23ba477ce5fa5403e96d9b8fe5c2f297e6421" gracePeriod=30
Sep 16 19:21:34.016099 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:34.015892    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kube-rbac-proxy" containerID="cri-o://87d642c5e9fb106b15203fb5c1d5d1e49377c6ec0ba10bd04f24c09af16ca198" gracePeriod=30
Sep 16 19:21:34.788276 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:34.788229    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 16 19:21:34.792554 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:34.792528    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:21:35.079279 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:35.079185    2565 generic.go:356] "Generic (PLEG): container finished" podID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerID="87d642c5e9fb106b15203fb5c1d5d1e49377c6ec0ba10bd04f24c09af16ca198" exitCode=2
Sep 16 19:21:35.079279 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:35.079256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" event={"ID":"dd1e16e1-229b-4334-94bd-8fc1a59f40f3","Type":"ContainerDied","Data":"87d642c5e9fb106b15203fb5c1d5d1e49377c6ec0ba10bd04f24c09af16ca198"}
Sep 16 19:21:38.094359 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.094321    2565 generic.go:356] "Generic (PLEG): container finished" podID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerID="682fbe77fbc9fa7096e904b058f23ba477ce5fa5403e96d9b8fe5c2f297e6421" exitCode=0
Sep 16 19:21:38.094750 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.094391    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" event={"ID":"dd1e16e1-229b-4334-94bd-8fc1a59f40f3","Type":"ContainerDied","Data":"682fbe77fbc9fa7096e904b058f23ba477ce5fa5403e96d9b8fe5c2f297e6421"}
Sep 16 19:21:38.171309 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.171282    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:21:38.271148 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.271017    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-cabundle-cert\") pod \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") "
Sep 16 19:21:38.271148 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.271060    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxvq8\" (UniqueName: \"kubernetes.io/projected/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kube-api-access-mxvq8\") pod \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") "
Sep 16 19:21:38.271148 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.271097    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kserve-provision-location\") pod \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") "
Sep 16 19:21:38.271437 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.271165    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-proxy-tls\") pod \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") "
Sep 16 19:21:38.271437 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.271232    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kube-rbac-proxy-sar-config\") pod \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\" (UID: \"dd1e16e1-229b-4334-94bd-8fc1a59f40f3\") "
Sep 16 19:21:38.271549 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.271502    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd1e16e1-229b-4334-94bd-8fc1a59f40f3" (UID: "dd1e16e1-229b-4334-94bd-8fc1a59f40f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:38.271613 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.271557    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd1e16e1-229b-4334-94bd-8fc1a59f40f3" (UID: "dd1e16e1-229b-4334-94bd-8fc1a59f40f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:38.271667 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.271649    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd1e16e1-229b-4334-94bd-8fc1a59f40f3" (UID: "dd1e16e1-229b-4334-94bd-8fc1a59f40f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:38.273509 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.273484    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd1e16e1-229b-4334-94bd-8fc1a59f40f3" (UID: "dd1e16e1-229b-4334-94bd-8fc1a59f40f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:38.273509 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.273499    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kube-api-access-mxvq8" (OuterVolumeSpecName: "kube-api-access-mxvq8") pod "dd1e16e1-229b-4334-94bd-8fc1a59f40f3" (UID: "dd1e16e1-229b-4334-94bd-8fc1a59f40f3"). InnerVolumeSpecName "kube-api-access-mxvq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:38.372308 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.372268    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:38.372308 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.372304    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:38.372308 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.372314    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxvq8\" (UniqueName: \"kubernetes.io/projected/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kube-api-access-mxvq8\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:38.372549 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.372324    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:38.372549 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:38.372335    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd1e16e1-229b-4334-94bd-8fc1a59f40f3-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:21:39.100386 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.100303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk" event={"ID":"dd1e16e1-229b-4334-94bd-8fc1a59f40f3","Type":"ContainerDied","Data":"8a9648d2c4062bf05fe66a3f739cd1a6928150f971e75b443fd05019dc324a50"}
Sep 16 19:21:39.100386 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.100364    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"
Sep 16 19:21:39.100836 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.100368    2565 scope.go:117] "RemoveContainer" containerID="87d642c5e9fb106b15203fb5c1d5d1e49377c6ec0ba10bd04f24c09af16ca198"
Sep 16 19:21:39.108155 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.108137    2565 scope.go:117] "RemoveContainer" containerID="682fbe77fbc9fa7096e904b058f23ba477ce5fa5403e96d9b8fe5c2f297e6421"
Sep 16 19:21:39.115652 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.115635    2565 scope.go:117] "RemoveContainer" containerID="6c2ed34a8e0e6a8e9ecbebf27204cdebd0a134566958d630974289f05d9570d7"
Sep 16 19:21:39.123850 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.123830    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"]
Sep 16 19:21:39.125583 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.125565    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-4hfwk"]
Sep 16 19:21:39.229684 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.229651    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"]
Sep 16 19:21:39.229914 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.229902    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="storage-initializer"
Sep 16 19:21:39.229958 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.229916    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="storage-initializer"
Sep 16 19:21:39.229958 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.229926    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kube-rbac-proxy"
Sep 16 19:21:39.229958 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.229933    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kube-rbac-proxy"
Sep 16 19:21:39.229958 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.229947    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="storage-initializer"
Sep 16 19:21:39.229958 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.229955    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="storage-initializer"
Sep 16 19:21:39.230183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.229966    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container"
Sep 16 19:21:39.230183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.229975    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container"
Sep 16 19:21:39.230183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.229989    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kube-rbac-proxy"
Sep 16 19:21:39.230183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.229994    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kube-rbac-proxy"
Sep 16 19:21:39.230183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.230001    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container"
Sep 16 19:21:39.230183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.230006    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container"
Sep 16 19:21:39.230183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.230049    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kube-rbac-proxy"
Sep 16 19:21:39.230183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.230056    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kserve-container"
Sep 16 19:21:39.230183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.230063    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="75381b4a-92fc-44ff-9e09-c8d72020b2ad" containerName="kube-rbac-proxy"
Sep 16 19:21:39.230183 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.230069    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" containerName="kserve-container"
Sep 16 19:21:39.234524 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.234507    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.237726 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.237704    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 16 19:21:39.237916 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.237893    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:21:39.238012 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.237896    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:21:39.238012 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.237992    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:21:39.238208 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.238146    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:21:39.238208 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.238202    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:21:39.238318 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.238235    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:21:39.245769 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.245743    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"]
Sep 16 19:21:39.279627 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.279591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.279833 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.279632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.279833 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.279734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.279833 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.279757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.279833 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.279796    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6htgl\" (UniqueName: \"kubernetes.io/projected/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kube-api-access-6htgl\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.381188 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.381057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6htgl\" (UniqueName: \"kubernetes.io/projected/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kube-api-access-6htgl\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.381188 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.381100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.381188 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.381150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.381477 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.381209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.381477 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.381226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.381477 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:21:39.381299    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:21:39.381477 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:21:39.381389    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-proxy-tls podName:dbe11a1c-0442-4aa3-bd7d-b9597549f4ff nodeName:}" failed. No retries permitted until 2026-09-16 19:21:39.88136687 +0000 UTC m=+2183.694955729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" (UID: "dbe11a1c-0442-4aa3-bd7d-b9597549f4ff") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:21:39.381705 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.381687    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.382071 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.382050    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.382129 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.382050    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.391258 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.391237    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6htgl\" (UniqueName: \"kubernetes.io/projected/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kube-api-access-6htgl\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.885473 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.885413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:39.887982 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:39.887947    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:40.145134 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:40.145015    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:40.430847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:40.430720    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"]
Sep 16 19:21:40.433454 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:21:40.433419    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe11a1c_0442_4aa3_bd7d_b9597549f4ff.slice/crio-47cd889de421a3d46e96a32168da87549c74230a921d1a91addb26ea0561e6cc WatchSource:0}: Error finding container 47cd889de421a3d46e96a32168da87549c74230a921d1a91addb26ea0561e6cc: Status 404 returned error can't find the container with id 47cd889de421a3d46e96a32168da87549c74230a921d1a91addb26ea0561e6cc
Sep 16 19:21:40.752332 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:40.752287    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd1e16e1-229b-4334-94bd-8fc1a59f40f3" path="/var/lib/kubelet/pods/dd1e16e1-229b-4334-94bd-8fc1a59f40f3/volumes"
Sep 16 19:21:41.112314 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:41.112207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" event={"ID":"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff","Type":"ContainerStarted","Data":"bcf904f0560e1bc58f8aa396adad59b4d9942ddf53b6175dd5c0ed131248b937"}
Sep 16 19:21:41.112314 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:41.112276    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" event={"ID":"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff","Type":"ContainerStarted","Data":"47cd889de421a3d46e96a32168da87549c74230a921d1a91addb26ea0561e6cc"}
Sep 16 19:21:45.128256 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:45.128219    2565 generic.go:356] "Generic (PLEG): container finished" podID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerID="bcf904f0560e1bc58f8aa396adad59b4d9942ddf53b6175dd5c0ed131248b937" exitCode=0
Sep 16 19:21:45.128728 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:45.128302    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" event={"ID":"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff","Type":"ContainerDied","Data":"bcf904f0560e1bc58f8aa396adad59b4d9942ddf53b6175dd5c0ed131248b937"}
Sep 16 19:21:46.135203 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:46.135165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" event={"ID":"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff","Type":"ContainerStarted","Data":"283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c"}
Sep 16 19:21:46.135679 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:46.135239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" event={"ID":"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff","Type":"ContainerStarted","Data":"7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271"}
Sep 16 19:21:46.135679 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:46.135457    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:46.135679 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:46.135501    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:21:46.163641 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:46.163594    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" podStartSLOduration=3.031500974 podStartE2EDuration="7.163581596s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.132080622s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:46.16183095 +0000 UTC m=+2189.975419809" watchObservedRunningTime="2026-09-16 19:21:46.163581596 +0000 UTC m=+2189.977170456"
Sep 16 19:21:52.144738 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:21:52.144706    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:22:22.150041 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:22.150004    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:22:29.263727 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:29.263693    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"]
Sep 16 19:22:29.264152 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:29.263964    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" podUID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerName="kserve-container" containerID="cri-o://7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271" gracePeriod=30
Sep 16 19:22:29.264152 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:29.264000    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" podUID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerName="kube-rbac-proxy" containerID="cri-o://283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c" gracePeriod=30
Sep 16 19:22:30.304904 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:30.304868    2565 generic.go:356] "Generic (PLEG): container finished" podID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerID="283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c" exitCode=2
Sep 16 19:22:30.305306 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:30.304938    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" event={"ID":"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff","Type":"ContainerDied","Data":"283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c"}
Sep 16 19:22:32.140250 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:32.140194    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" podUID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 19:22:34.503735 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.503699    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"]
Sep 16 19:22:34.507077 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.507059    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.511414 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.511385    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 16 19:22:34.511804 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.511391    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:22:34.527197 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.527168    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"]
Sep 16 19:22:34.637007 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.636942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d722986-342e-4413-924b-a3b121211022-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.637285 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.637022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d722986-342e-4413-924b-a3b121211022-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.637285 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.637081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d722986-342e-4413-924b-a3b121211022-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.637285 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.637194    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d722986-342e-4413-924b-a3b121211022-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.637285 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.637252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dhft\" (UniqueName: \"kubernetes.io/projected/0d722986-342e-4413-924b-a3b121211022-kube-api-access-8dhft\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.738157 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.738100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d722986-342e-4413-924b-a3b121211022-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.738157 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.738163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d722986-342e-4413-924b-a3b121211022-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.738443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.738195    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d722986-342e-4413-924b-a3b121211022-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.738443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.738223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d722986-342e-4413-924b-a3b121211022-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.738443 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.738254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dhft\" (UniqueName: \"kubernetes.io/projected/0d722986-342e-4413-924b-a3b121211022-kube-api-access-8dhft\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.738709 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.738681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d722986-342e-4413-924b-a3b121211022-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.738935 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.738912    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d722986-342e-4413-924b-a3b121211022-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.739000 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.738946    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d722986-342e-4413-924b-a3b121211022-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.740792 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.740774    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d722986-342e-4413-924b-a3b121211022-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.747522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.747487    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dhft\" (UniqueName: \"kubernetes.io/projected/0d722986-342e-4413-924b-a3b121211022-kube-api-access-8dhft\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:34.822076 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:34.821966    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:35.103946 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:35.103841    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"]
Sep 16 19:22:35.106870 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:22:35.106841    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d722986_342e_4413_924b_a3b121211022.slice/crio-c440b83193392bf013ef80eb031a3519f72b5f335a53e4dd3cb25eeb9eff3638 WatchSource:0}: Error finding container c440b83193392bf013ef80eb031a3519f72b5f335a53e4dd3cb25eeb9eff3638: Status 404 returned error can't find the container with id c440b83193392bf013ef80eb031a3519f72b5f335a53e4dd3cb25eeb9eff3638
Sep 16 19:22:35.323651 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:35.323615    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" event={"ID":"0d722986-342e-4413-924b-a3b121211022","Type":"ContainerStarted","Data":"41dd544fd76149c33d11416d3eaa52ec17ece05103ce68da5c26c4472f336164"}
Sep 16 19:22:35.323831 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:35.323673    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" event={"ID":"0d722986-342e-4413-924b-a3b121211022","Type":"ContainerStarted","Data":"c440b83193392bf013ef80eb031a3519f72b5f335a53e4dd3cb25eeb9eff3638"}
Sep 16 19:22:36.712932 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.712900    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:22:36.857075 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.856976    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6htgl\" (UniqueName: \"kubernetes.io/projected/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kube-api-access-6htgl\") pod \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") "
Sep 16 19:22:36.857075 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.857043    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kserve-provision-location\") pod \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") "
Sep 16 19:22:36.857339 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.857086    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-proxy-tls\") pod \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") "
Sep 16 19:22:36.857339 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.857133    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-cabundle-cert\") pod \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") "
Sep 16 19:22:36.857339 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.857159    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kube-rbac-proxy-sar-config\") pod \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\" (UID: \"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff\") "
Sep 16 19:22:36.857497 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.857454    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" (UID: "dbe11a1c-0442-4aa3-bd7d-b9597549f4ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:36.857554 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.857517    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" (UID: "dbe11a1c-0442-4aa3-bd7d-b9597549f4ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:36.857554 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.857538    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" (UID: "dbe11a1c-0442-4aa3-bd7d-b9597549f4ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:36.859237 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.859212    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" (UID: "dbe11a1c-0442-4aa3-bd7d-b9597549f4ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:36.859353 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.859212    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kube-api-access-6htgl" (OuterVolumeSpecName: "kube-api-access-6htgl") pod "dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" (UID: "dbe11a1c-0442-4aa3-bd7d-b9597549f4ff"). InnerVolumeSpecName "kube-api-access-6htgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:36.957974 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.957930    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6htgl\" (UniqueName: \"kubernetes.io/projected/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kube-api-access-6htgl\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:22:36.957974 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.957965    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:22:36.957974 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.957975    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:22:36.957974 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.957986    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:22:36.958269 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:36.957996    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:22:37.333777 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.333738    2565 generic.go:356] "Generic (PLEG): container finished" podID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerID="7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271" exitCode=0
Sep 16 19:22:37.333969 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.333810    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"
Sep 16 19:22:37.333969 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.333819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" event={"ID":"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff","Type":"ContainerDied","Data":"7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271"}
Sep 16 19:22:37.333969 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.333883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2" event={"ID":"dbe11a1c-0442-4aa3-bd7d-b9597549f4ff","Type":"ContainerDied","Data":"47cd889de421a3d46e96a32168da87549c74230a921d1a91addb26ea0561e6cc"}
Sep 16 19:22:37.333969 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.333910    2565 scope.go:117] "RemoveContainer" containerID="283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c"
Sep 16 19:22:37.343034 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.343013    2565 scope.go:117] "RemoveContainer" containerID="7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271"
Sep 16 19:22:37.351184 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.351155    2565 scope.go:117] "RemoveContainer" containerID="bcf904f0560e1bc58f8aa396adad59b4d9942ddf53b6175dd5c0ed131248b937"
Sep 16 19:22:37.357869 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.357839    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"]
Sep 16 19:22:37.360170 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.360146    2565 scope.go:117] "RemoveContainer" containerID="283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c"
Sep 16 19:22:37.360710 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:22:37.360670    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c\": container with ID starting with 283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c not found: ID does not exist" containerID="283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c"
Sep 16 19:22:37.360841 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.360723    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c"} err="failed to get container status \"283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c\": rpc error: code = NotFound desc = could not find container \"283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c\": container with ID starting with 283ed164546f38e416e4dce44ae5d05049c8c29ae05364bafc9a0484f80cb50c not found: ID does not exist"
Sep 16 19:22:37.360841 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.360748    2565 scope.go:117] "RemoveContainer" containerID="7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271"
Sep 16 19:22:37.361149 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:22:37.361118    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271\": container with ID starting with 7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271 not found: ID does not exist" containerID="7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271"
Sep 16 19:22:37.361257 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.361156    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271"} err="failed to get container status \"7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271\": rpc error: code = NotFound desc = could not find container \"7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271\": container with ID starting with 7b61077fe9af2b740c75fd70a38de709547584fd86d1168e06b2ca805a339271 not found: ID does not exist"
Sep 16 19:22:37.361257 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.361179    2565 scope.go:117] "RemoveContainer" containerID="bcf904f0560e1bc58f8aa396adad59b4d9942ddf53b6175dd5c0ed131248b937"
Sep 16 19:22:37.361490 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:22:37.361472    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf904f0560e1bc58f8aa396adad59b4d9942ddf53b6175dd5c0ed131248b937\": container with ID starting with bcf904f0560e1bc58f8aa396adad59b4d9942ddf53b6175dd5c0ed131248b937 not found: ID does not exist" containerID="bcf904f0560e1bc58f8aa396adad59b4d9942ddf53b6175dd5c0ed131248b937"
Sep 16 19:22:37.361538 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.361498    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf904f0560e1bc58f8aa396adad59b4d9942ddf53b6175dd5c0ed131248b937"} err="failed to get container status \"bcf904f0560e1bc58f8aa396adad59b4d9942ddf53b6175dd5c0ed131248b937\": rpc error: code = NotFound desc = could not find container \"bcf904f0560e1bc58f8aa396adad59b4d9942ddf53b6175dd5c0ed131248b937\": container with ID starting with bcf904f0560e1bc58f8aa396adad59b4d9942ddf53b6175dd5c0ed131248b937 not found: ID does not exist"
Sep 16 19:22:37.362209 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:37.362191    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-nxgx2"]
Sep 16 19:22:38.751544 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:38.751511    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" path="/var/lib/kubelet/pods/dbe11a1c-0442-4aa3-bd7d-b9597549f4ff/volumes"
Sep 16 19:22:40.347630 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:40.347594    2565 generic.go:356] "Generic (PLEG): container finished" podID="0d722986-342e-4413-924b-a3b121211022" containerID="41dd544fd76149c33d11416d3eaa52ec17ece05103ce68da5c26c4472f336164" exitCode=0
Sep 16 19:22:40.348028 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:40.347653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" event={"ID":"0d722986-342e-4413-924b-a3b121211022","Type":"ContainerDied","Data":"41dd544fd76149c33d11416d3eaa52ec17ece05103ce68da5c26c4472f336164"}
Sep 16 19:22:41.354230 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:41.354193    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" event={"ID":"0d722986-342e-4413-924b-a3b121211022","Type":"ContainerStarted","Data":"4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430"}
Sep 16 19:22:41.354713 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:41.354269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" event={"ID":"0d722986-342e-4413-924b-a3b121211022","Type":"ContainerStarted","Data":"dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635"}
Sep 16 19:22:41.354713 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:41.354479    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:41.354713 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:41.354522    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:22:41.382000 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:41.381946    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" podStartSLOduration=3.138281219 podStartE2EDuration="7.38192815s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.243646931s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:41.380200558 +0000 UTC m=+2245.193789418" watchObservedRunningTime="2026-09-16 19:22:41.38192815 +0000 UTC m=+2245.195517009"
Sep 16 19:22:47.364055 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:22:47.364023    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:23:17.446953 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:17.446919    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:23:24.503433 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:24.503394    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"]
Sep 16 19:23:24.504543 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:24.504484    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" podUID="0d722986-342e-4413-924b-a3b121211022" containerName="kserve-container" containerID="cri-o://dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635" gracePeriod=30
Sep 16 19:23:24.505140 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:24.505085    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" podUID="0d722986-342e-4413-924b-a3b121211022" containerName="kube-rbac-proxy" containerID="cri-o://4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430" gracePeriod=30
Sep 16 19:23:25.533831 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:25.533794    2565 generic.go:356] "Generic (PLEG): container finished" podID="0d722986-342e-4413-924b-a3b121211022" containerID="4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430" exitCode=2
Sep 16 19:23:25.534256 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:25.533846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" event={"ID":"0d722986-342e-4413-924b-a3b121211022","Type":"ContainerDied","Data":"4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430"}
Sep 16 19:23:27.359091 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:27.359031    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" podUID="0d722986-342e-4413-924b-a3b121211022" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 16 19:23:27.365242 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:27.365210    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" podUID="0d722986-342e-4413-924b-a3b121211022" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:23:29.755631 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.755590    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"]
Sep 16 19:23:29.756179 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.755998    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerName="storage-initializer"
Sep 16 19:23:29.756179 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.756018    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerName="storage-initializer"
Sep 16 19:23:29.756179 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.756035    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerName="kube-rbac-proxy"
Sep 16 19:23:29.756179 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.756043    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerName="kube-rbac-proxy"
Sep 16 19:23:29.756179 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.756055    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerName="kserve-container"
Sep 16 19:23:29.756179 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.756063    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerName="kserve-container"
Sep 16 19:23:29.756179 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.756148    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerName="kube-rbac-proxy"
Sep 16 19:23:29.756179 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.756162    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe11a1c-0442-4aa3-bd7d-b9597549f4ff" containerName="kserve-container"
Sep 16 19:23:29.759258 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.759236    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.761786 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.761763    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 16 19:23:29.762038 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.762022    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:23:29.770078 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.770055    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"]
Sep 16 19:23:29.864941 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.864900    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5929ac1-ac6a-464b-b909-a549db6fa640-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.865181 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.864957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5929ac1-ac6a-464b-b909-a549db6fa640-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.865181 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.865017    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhzf2\" (UniqueName: \"kubernetes.io/projected/d5929ac1-ac6a-464b-b909-a549db6fa640-kube-api-access-dhzf2\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.865181 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.865067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5929ac1-ac6a-464b-b909-a549db6fa640-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.865181 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.865151    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5929ac1-ac6a-464b-b909-a549db6fa640-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.965788 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.965751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5929ac1-ac6a-464b-b909-a549db6fa640-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.965981 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.965797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5929ac1-ac6a-464b-b909-a549db6fa640-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.965981 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.965839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5929ac1-ac6a-464b-b909-a549db6fa640-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.965981 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.965863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhzf2\" (UniqueName: \"kubernetes.io/projected/d5929ac1-ac6a-464b-b909-a549db6fa640-kube-api-access-dhzf2\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.965981 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.965893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5929ac1-ac6a-464b-b909-a549db6fa640-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.966365 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.966343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5929ac1-ac6a-464b-b909-a549db6fa640-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.966529 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.966510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5929ac1-ac6a-464b-b909-a549db6fa640-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.966592 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.966518    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5929ac1-ac6a-464b-b909-a549db6fa640-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.968520 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.968499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5929ac1-ac6a-464b-b909-a549db6fa640-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:29.974932 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:29.974900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhzf2\" (UniqueName: \"kubernetes.io/projected/d5929ac1-ac6a-464b-b909-a549db6fa640-kube-api-access-dhzf2\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vndc7\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:30.069244 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:30.069159    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:30.352407 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:30.352378    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"]
Sep 16 19:23:30.355386 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:23:30.355346    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5929ac1_ac6a_464b_b909_a549db6fa640.slice/crio-9fa258678a1518a8975c0cf5e851502dc2d77ba5e134c7e312a20f172169ffed WatchSource:0}: Error finding container 9fa258678a1518a8975c0cf5e851502dc2d77ba5e134c7e312a20f172169ffed: Status 404 returned error can't find the container with id 9fa258678a1518a8975c0cf5e851502dc2d77ba5e134c7e312a20f172169ffed
Sep 16 19:23:30.553790 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:30.553753    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" event={"ID":"d5929ac1-ac6a-464b-b909-a549db6fa640","Type":"ContainerStarted","Data":"4fd07f36b213cf5c1043908d574ee6228664e0e8bc58b1eadf89341f3b8d695f"}
Sep 16 19:23:30.553980 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:30.553815    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" event={"ID":"d5929ac1-ac6a-464b-b909-a549db6fa640","Type":"ContainerStarted","Data":"9fa258678a1518a8975c0cf5e851502dc2d77ba5e134c7e312a20f172169ffed"}
Sep 16 19:23:31.857634 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:31.857609    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:23:31.985399 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:31.985369    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d722986-342e-4413-924b-a3b121211022-proxy-tls\") pod \"0d722986-342e-4413-924b-a3b121211022\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") "
Sep 16 19:23:31.985581 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:31.985410    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dhft\" (UniqueName: \"kubernetes.io/projected/0d722986-342e-4413-924b-a3b121211022-kube-api-access-8dhft\") pod \"0d722986-342e-4413-924b-a3b121211022\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") "
Sep 16 19:23:31.985581 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:31.985452    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d722986-342e-4413-924b-a3b121211022-cabundle-cert\") pod \"0d722986-342e-4413-924b-a3b121211022\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") "
Sep 16 19:23:31.985581 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:31.985479    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d722986-342e-4413-924b-a3b121211022-kube-rbac-proxy-sar-config\") pod \"0d722986-342e-4413-924b-a3b121211022\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") "
Sep 16 19:23:31.985581 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:31.985499    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d722986-342e-4413-924b-a3b121211022-kserve-provision-location\") pod \"0d722986-342e-4413-924b-a3b121211022\" (UID: \"0d722986-342e-4413-924b-a3b121211022\") "
Sep 16 19:23:31.985880 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:31.985857    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d722986-342e-4413-924b-a3b121211022-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d722986-342e-4413-924b-a3b121211022" (UID: "0d722986-342e-4413-924b-a3b121211022"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:31.985880 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:31.985868    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d722986-342e-4413-924b-a3b121211022-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d722986-342e-4413-924b-a3b121211022" (UID: "0d722986-342e-4413-924b-a3b121211022"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:31.985976 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:31.985879    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d722986-342e-4413-924b-a3b121211022-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d722986-342e-4413-924b-a3b121211022" (UID: "0d722986-342e-4413-924b-a3b121211022"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:31.987531 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:31.987509    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d722986-342e-4413-924b-a3b121211022-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d722986-342e-4413-924b-a3b121211022" (UID: "0d722986-342e-4413-924b-a3b121211022"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:31.987640 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:31.987624    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d722986-342e-4413-924b-a3b121211022-kube-api-access-8dhft" (OuterVolumeSpecName: "kube-api-access-8dhft") pod "0d722986-342e-4413-924b-a3b121211022" (UID: "0d722986-342e-4413-924b-a3b121211022"). InnerVolumeSpecName "kube-api-access-8dhft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:32.086003 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.085968    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d722986-342e-4413-924b-a3b121211022-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:32.086003 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.086002    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d722986-342e-4413-924b-a3b121211022-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:32.086003 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.086012    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d722986-342e-4413-924b-a3b121211022-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:32.086342 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.086022    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d722986-342e-4413-924b-a3b121211022-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:32.086342 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.086032    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dhft\" (UniqueName: \"kubernetes.io/projected/0d722986-342e-4413-924b-a3b121211022-kube-api-access-8dhft\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:23:32.563455 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.563411    2565 generic.go:356] "Generic (PLEG): container finished" podID="0d722986-342e-4413-924b-a3b121211022" containerID="dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635" exitCode=0
Sep 16 19:23:32.563673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.563474    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"
Sep 16 19:23:32.563673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.563486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" event={"ID":"0d722986-342e-4413-924b-a3b121211022","Type":"ContainerDied","Data":"dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635"}
Sep 16 19:23:32.563673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.563543    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh" event={"ID":"0d722986-342e-4413-924b-a3b121211022","Type":"ContainerDied","Data":"c440b83193392bf013ef80eb031a3519f72b5f335a53e4dd3cb25eeb9eff3638"}
Sep 16 19:23:32.563673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.563572    2565 scope.go:117] "RemoveContainer" containerID="4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430"
Sep 16 19:23:32.572493 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.572470    2565 scope.go:117] "RemoveContainer" containerID="dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635"
Sep 16 19:23:32.580123 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.580086    2565 scope.go:117] "RemoveContainer" containerID="41dd544fd76149c33d11416d3eaa52ec17ece05103ce68da5c26c4472f336164"
Sep 16 19:23:32.586839 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.586816    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"]
Sep 16 19:23:32.588012 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.587987    2565 scope.go:117] "RemoveContainer" containerID="4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430"
Sep 16 19:23:32.588326 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:23:32.588309    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430\": container with ID starting with 4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430 not found: ID does not exist" containerID="4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430"
Sep 16 19:23:32.588377 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.588338    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430"} err="failed to get container status \"4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430\": rpc error: code = NotFound desc = could not find container \"4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430\": container with ID starting with 4ff7f517cec0c57f422e4248470bc331508834f15551e8f829e8714d83f06430 not found: ID does not exist"
Sep 16 19:23:32.588377 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.588357    2565 scope.go:117] "RemoveContainer" containerID="dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635"
Sep 16 19:23:32.588590 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:23:32.588574    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635\": container with ID starting with dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635 not found: ID does not exist" containerID="dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635"
Sep 16 19:23:32.588627 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.588594    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635"} err="failed to get container status \"dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635\": rpc error: code = NotFound desc = could not find container \"dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635\": container with ID starting with dc96e025dde058d4843835d055a940783f39ff63a7a9c97ba412fee3474cb635 not found: ID does not exist"
Sep 16 19:23:32.588627 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.588607    2565 scope.go:117] "RemoveContainer" containerID="41dd544fd76149c33d11416d3eaa52ec17ece05103ce68da5c26c4472f336164"
Sep 16 19:23:32.588883 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:23:32.588862    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41dd544fd76149c33d11416d3eaa52ec17ece05103ce68da5c26c4472f336164\": container with ID starting with 41dd544fd76149c33d11416d3eaa52ec17ece05103ce68da5c26c4472f336164 not found: ID does not exist" containerID="41dd544fd76149c33d11416d3eaa52ec17ece05103ce68da5c26c4472f336164"
Sep 16 19:23:32.588930 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.588890    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41dd544fd76149c33d11416d3eaa52ec17ece05103ce68da5c26c4472f336164"} err="failed to get container status \"41dd544fd76149c33d11416d3eaa52ec17ece05103ce68da5c26c4472f336164\": rpc error: code = NotFound desc = could not find container \"41dd544fd76149c33d11416d3eaa52ec17ece05103ce68da5c26c4472f336164\": container with ID starting with 41dd544fd76149c33d11416d3eaa52ec17ece05103ce68da5c26c4472f336164 not found: ID does not exist"
Sep 16 19:23:32.591068 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.591050    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-c9gjh"]
Sep 16 19:23:32.751055 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:32.751019    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d722986-342e-4413-924b-a3b121211022" path="/var/lib/kubelet/pods/0d722986-342e-4413-924b-a3b121211022/volumes"
Sep 16 19:23:34.573884 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:34.573847    2565 generic.go:356] "Generic (PLEG): container finished" podID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerID="4fd07f36b213cf5c1043908d574ee6228664e0e8bc58b1eadf89341f3b8d695f" exitCode=0
Sep 16 19:23:34.574313 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:34.573900    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" event={"ID":"d5929ac1-ac6a-464b-b909-a549db6fa640","Type":"ContainerDied","Data":"4fd07f36b213cf5c1043908d574ee6228664e0e8bc58b1eadf89341f3b8d695f"}
Sep 16 19:23:35.579361 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:35.579328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" event={"ID":"d5929ac1-ac6a-464b-b909-a549db6fa640","Type":"ContainerStarted","Data":"43b4c3ec540f06d99b0f9c0b2bf4f341173a36e1accd8544a4e633ac64304c1a"}
Sep 16 19:23:35.579792 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:35.579390    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" event={"ID":"d5929ac1-ac6a-464b-b909-a549db6fa640","Type":"ContainerStarted","Data":"8409ac5eab6db3e6b7eabf800180e89e4d92928232f1bd0c70a38e8b631a3ac4"}
Sep 16 19:23:35.579792 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:35.579666    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:35.579900 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:35.579826    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:35.581376 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:35.581334    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:23:35.605923 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:35.605871    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podStartSLOduration=2.929660319 podStartE2EDuration="6.605855563s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.676195244s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:35.605173295 +0000 UTC m=+2299.418762153" watchObservedRunningTime="2026-09-16 19:23:35.605855563 +0000 UTC m=+2299.419444413"
Sep 16 19:23:36.583578 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:36.583536    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:23:41.591373 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:41.591343    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:23:41.591901 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:41.591867    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:23:51.591990 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:23:51.591949    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:24:01.592257 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:01.592208    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:24:11.592797 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:11.592699    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:24:21.591903 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:21.591857    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:24:29.550486 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.550448    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"]
Sep 16 19:24:29.550863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.550720    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d722986-342e-4413-924b-a3b121211022" containerName="storage-initializer"
Sep 16 19:24:29.550863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.550730    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d722986-342e-4413-924b-a3b121211022" containerName="storage-initializer"
Sep 16 19:24:29.550863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.550741    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d722986-342e-4413-924b-a3b121211022" containerName="kube-rbac-proxy"
Sep 16 19:24:29.550863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.550746    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d722986-342e-4413-924b-a3b121211022" containerName="kube-rbac-proxy"
Sep 16 19:24:29.550863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.550754    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d722986-342e-4413-924b-a3b121211022" containerName="kserve-container"
Sep 16 19:24:29.550863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.550760    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d722986-342e-4413-924b-a3b121211022" containerName="kserve-container"
Sep 16 19:24:29.550863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.550811    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d722986-342e-4413-924b-a3b121211022" containerName="kserve-container"
Sep 16 19:24:29.550863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.550819    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d722986-342e-4413-924b-a3b121211022" containerName="kube-rbac-proxy"
Sep 16 19:24:29.553774 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.553757    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.556838 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.556818    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 16 19:24:29.556838 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.556818    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:24:29.568391 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.568364    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"]
Sep 16 19:24:29.650994 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.650958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.650994 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.650998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.651277 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.651036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.651277 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.651166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.651277 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.651199    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfx8j\" (UniqueName: \"kubernetes.io/projected/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kube-api-access-nfx8j\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.751886 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.751844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfx8j\" (UniqueName: \"kubernetes.io/projected/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kube-api-access-nfx8j\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.752084 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.751898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.752084 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.751948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.752084 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.751989    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.752084 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.752056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.752344 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:24:29.752183    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 16 19:24:29.752344 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:24:29.752257    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-proxy-tls podName:2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f nodeName:}" failed. No retries permitted until 2026-09-16 19:24:30.252236839 +0000 UTC m=+2354.065825678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl" (UID: "2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 16 19:24:29.752460 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.752406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.752689 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.752670    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.752780 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.752763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:29.761221 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:29.761204    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfx8j\" (UniqueName: \"kubernetes.io/projected/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kube-api-access-nfx8j\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:30.256620 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:30.256560    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:30.259156 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:30.259100    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:30.463475 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:30.463435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:30.738148 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:30.738096    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"]
Sep 16 19:24:30.741878 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:24:30.741847    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcc41d5_b3a2_43e8_9d93_ea3b3f87e64f.slice/crio-63d443c98b1cf887b4fffea42835ae181dfb6314fdc1ce7585c0536c336e9727 WatchSource:0}: Error finding container 63d443c98b1cf887b4fffea42835ae181dfb6314fdc1ce7585c0536c336e9727: Status 404 returned error can't find the container with id 63d443c98b1cf887b4fffea42835ae181dfb6314fdc1ce7585c0536c336e9727
Sep 16 19:24:30.785879 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:30.785837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl" event={"ID":"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f","Type":"ContainerStarted","Data":"63d443c98b1cf887b4fffea42835ae181dfb6314fdc1ce7585c0536c336e9727"}
Sep 16 19:24:31.592793 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:31.592745    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:24:31.797780 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:31.797743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl" event={"ID":"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f","Type":"ContainerStarted","Data":"270a0ad583615b15f8f1abf978d2211b36e235addfa0ce26c9287684487a2b3f"}
Sep 16 19:24:34.810711 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:34.810676    2565 generic.go:356] "Generic (PLEG): container finished" podID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerID="270a0ad583615b15f8f1abf978d2211b36e235addfa0ce26c9287684487a2b3f" exitCode=0
Sep 16 19:24:34.811265 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:34.810731    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl" event={"ID":"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f","Type":"ContainerDied","Data":"270a0ad583615b15f8f1abf978d2211b36e235addfa0ce26c9287684487a2b3f"}
Sep 16 19:24:35.817470 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:35.817434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl" event={"ID":"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f","Type":"ContainerStarted","Data":"a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8"}
Sep 16 19:24:35.817860 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:35.817506    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl" event={"ID":"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f","Type":"ContainerStarted","Data":"fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10"}
Sep 16 19:24:35.817860 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:35.817775    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:35.840264 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:35.840213    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl" podStartSLOduration=2.934653987 podStartE2EDuration="6.840193719s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.905539732s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:35.839534837 +0000 UTC m=+2359.653123701" watchObservedRunningTime="2026-09-16 19:24:35.840193719 +0000 UTC m=+2359.653782578"
Sep 16 19:24:36.822121 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:36.822060    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:41.594362 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:41.594315    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:24:42.831448 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:42.831414    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:24:49.867081 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:49.867050    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"]
Sep 16 19:24:49.867561 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:49.867316    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container" containerID="cri-o://8409ac5eab6db3e6b7eabf800180e89e4d92928232f1bd0c70a38e8b631a3ac4" gracePeriod=30
Sep 16 19:24:49.867561 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:49.867345    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kube-rbac-proxy" containerID="cri-o://43b4c3ec540f06d99b0f9c0b2bf4f341173a36e1accd8544a4e633ac64304c1a" gracePeriod=30
Sep 16 19:24:50.874509 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:50.874473    2565 generic.go:356] "Generic (PLEG): container finished" podID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerID="43b4c3ec540f06d99b0f9c0b2bf4f341173a36e1accd8544a4e633ac64304c1a" exitCode=2
Sep 16 19:24:50.874891 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:50.874542    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" event={"ID":"d5929ac1-ac6a-464b-b909-a549db6fa640","Type":"ContainerDied","Data":"43b4c3ec540f06d99b0f9c0b2bf4f341173a36e1accd8544a4e633ac64304c1a"}
Sep 16 19:24:51.584578 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:51.584516    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 16 19:24:51.592048 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:51.592005    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:24:53.891516 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:53.891472    2565 generic.go:356] "Generic (PLEG): container finished" podID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerID="8409ac5eab6db3e6b7eabf800180e89e4d92928232f1bd0c70a38e8b631a3ac4" exitCode=0
Sep 16 19:24:53.891863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:53.891537    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" event={"ID":"d5929ac1-ac6a-464b-b909-a549db6fa640","Type":"ContainerDied","Data":"8409ac5eab6db3e6b7eabf800180e89e4d92928232f1bd0c70a38e8b631a3ac4"}
Sep 16 19:24:54.014925 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.014900    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:24:54.159847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.159674    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5929ac1-ac6a-464b-b909-a549db6fa640-kube-rbac-proxy-sar-config\") pod \"d5929ac1-ac6a-464b-b909-a549db6fa640\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") "
Sep 16 19:24:54.159847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.159759    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5929ac1-ac6a-464b-b909-a549db6fa640-kserve-provision-location\") pod \"d5929ac1-ac6a-464b-b909-a549db6fa640\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") "
Sep 16 19:24:54.159847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.159815    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5929ac1-ac6a-464b-b909-a549db6fa640-cabundle-cert\") pod \"d5929ac1-ac6a-464b-b909-a549db6fa640\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") "
Sep 16 19:24:54.160291 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.159859    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhzf2\" (UniqueName: \"kubernetes.io/projected/d5929ac1-ac6a-464b-b909-a549db6fa640-kube-api-access-dhzf2\") pod \"d5929ac1-ac6a-464b-b909-a549db6fa640\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") "
Sep 16 19:24:54.160291 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.159905    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5929ac1-ac6a-464b-b909-a549db6fa640-proxy-tls\") pod \"d5929ac1-ac6a-464b-b909-a549db6fa640\" (UID: \"d5929ac1-ac6a-464b-b909-a549db6fa640\") "
Sep 16 19:24:54.160291 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.160159    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5929ac1-ac6a-464b-b909-a549db6fa640-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5929ac1-ac6a-464b-b909-a549db6fa640" (UID: "d5929ac1-ac6a-464b-b909-a549db6fa640"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:54.160291 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.160246    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5929ac1-ac6a-464b-b909-a549db6fa640-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5929ac1-ac6a-464b-b909-a549db6fa640" (UID: "d5929ac1-ac6a-464b-b909-a549db6fa640"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:54.160451 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.160289    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5929ac1-ac6a-464b-b909-a549db6fa640-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5929ac1-ac6a-464b-b909-a549db6fa640" (UID: "d5929ac1-ac6a-464b-b909-a549db6fa640"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:54.160451 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.160299    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5929ac1-ac6a-464b-b909-a549db6fa640-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:24:54.162616 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.162570    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5929ac1-ac6a-464b-b909-a549db6fa640-kube-api-access-dhzf2" (OuterVolumeSpecName: "kube-api-access-dhzf2") pod "d5929ac1-ac6a-464b-b909-a549db6fa640" (UID: "d5929ac1-ac6a-464b-b909-a549db6fa640"). InnerVolumeSpecName "kube-api-access-dhzf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:54.162616 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.162586    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5929ac1-ac6a-464b-b909-a549db6fa640-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5929ac1-ac6a-464b-b909-a549db6fa640" (UID: "d5929ac1-ac6a-464b-b909-a549db6fa640"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:54.260738 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.260684    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5929ac1-ac6a-464b-b909-a549db6fa640-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:24:54.260738 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.260731    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5929ac1-ac6a-464b-b909-a549db6fa640-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:24:54.260738 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.260741    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5929ac1-ac6a-464b-b909-a549db6fa640-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:24:54.260738 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.260751    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhzf2\" (UniqueName: \"kubernetes.io/projected/d5929ac1-ac6a-464b-b909-a549db6fa640-kube-api-access-dhzf2\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:24:54.897267 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.897229    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7" event={"ID":"d5929ac1-ac6a-464b-b909-a549db6fa640","Type":"ContainerDied","Data":"9fa258678a1518a8975c0cf5e851502dc2d77ba5e134c7e312a20f172169ffed"}
Sep 16 19:24:54.897728 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.897314    2565 scope.go:117] "RemoveContainer" containerID="43b4c3ec540f06d99b0f9c0b2bf4f341173a36e1accd8544a4e633ac64304c1a"
Sep 16 19:24:54.897728 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.897250    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"
Sep 16 19:24:54.905554 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.905532    2565 scope.go:117] "RemoveContainer" containerID="8409ac5eab6db3e6b7eabf800180e89e4d92928232f1bd0c70a38e8b631a3ac4"
Sep 16 19:24:54.913363 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.913331    2565 scope.go:117] "RemoveContainer" containerID="4fd07f36b213cf5c1043908d574ee6228664e0e8bc58b1eadf89341f3b8d695f"
Sep 16 19:24:54.920788 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.920766    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"]
Sep 16 19:24:54.926535 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.926512    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vndc7"]
Sep 16 19:24:54.988571 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.988534    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"]
Sep 16 19:24:54.988813 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.988802    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kube-rbac-proxy"
Sep 16 19:24:54.988882 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.988815    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kube-rbac-proxy"
Sep 16 19:24:54.988882 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.988834    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container"
Sep 16 19:24:54.988882 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.988840    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container"
Sep 16 19:24:54.988882 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.988847    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="storage-initializer"
Sep 16 19:24:54.988882 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.988852    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="storage-initializer"
Sep 16 19:24:54.989057 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.988900    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kserve-container"
Sep 16 19:24:54.989057 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.988908    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" containerName="kube-rbac-proxy"
Sep 16 19:24:54.993244 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.993224    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:54.996181 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.996158    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:24:54.996329 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:54.996224    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 16 19:24:55.004157 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.004097    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"]
Sep 16 19:24:55.069325 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.069222    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53506d68-8ec9-48cc-9425-a127d9c8df28-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.069325 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.069291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhcrb\" (UniqueName: \"kubernetes.io/projected/53506d68-8ec9-48cc-9425-a127d9c8df28-kube-api-access-jhcrb\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.069325 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.069313    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53506d68-8ec9-48cc-9425-a127d9c8df28-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.069583 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.069342    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53506d68-8ec9-48cc-9425-a127d9c8df28-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.069583 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.069421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53506d68-8ec9-48cc-9425-a127d9c8df28-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.170415 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.170375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53506d68-8ec9-48cc-9425-a127d9c8df28-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.170415 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.170420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53506d68-8ec9-48cc-9425-a127d9c8df28-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.170657 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.170466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhcrb\" (UniqueName: \"kubernetes.io/projected/53506d68-8ec9-48cc-9425-a127d9c8df28-kube-api-access-jhcrb\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.170657 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.170488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53506d68-8ec9-48cc-9425-a127d9c8df28-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.170657 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.170514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53506d68-8ec9-48cc-9425-a127d9c8df28-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.170966 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.170941    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53506d68-8ec9-48cc-9425-a127d9c8df28-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.171261 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.171242    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53506d68-8ec9-48cc-9425-a127d9c8df28-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.171310 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.171286    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53506d68-8ec9-48cc-9425-a127d9c8df28-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.172965 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.172941    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53506d68-8ec9-48cc-9425-a127d9c8df28-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.180001 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.179972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhcrb\" (UniqueName: \"kubernetes.io/projected/53506d68-8ec9-48cc-9425-a127d9c8df28-kube-api-access-jhcrb\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.303099 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.303038    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:24:55.575882 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.575858    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"]
Sep 16 19:24:55.577497 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:24:55.577465    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53506d68_8ec9_48cc_9425_a127d9c8df28.slice/crio-fad2ed161a8c10023455f143d5ce5bb871c36bfc34846a7111d3c3f153041ef6 WatchSource:0}: Error finding container fad2ed161a8c10023455f143d5ce5bb871c36bfc34846a7111d3c3f153041ef6: Status 404 returned error can't find the container with id fad2ed161a8c10023455f143d5ce5bb871c36bfc34846a7111d3c3f153041ef6
Sep 16 19:24:55.903737 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.903643    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" event={"ID":"53506d68-8ec9-48cc-9425-a127d9c8df28","Type":"ContainerStarted","Data":"14023920013a49e0e5b8695410d2770cf5d6264a21f66a9473567559748c4641"}
Sep 16 19:24:55.903737 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:55.903714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" event={"ID":"53506d68-8ec9-48cc-9425-a127d9c8df28","Type":"ContainerStarted","Data":"fad2ed161a8c10023455f143d5ce5bb871c36bfc34846a7111d3c3f153041ef6"}
Sep 16 19:24:56.752995 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:56.752958    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5929ac1-ac6a-464b-b909-a549db6fa640" path="/var/lib/kubelet/pods/d5929ac1-ac6a-464b-b909-a549db6fa640/volumes"
Sep 16 19:24:59.919912 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:59.919877    2565 generic.go:356] "Generic (PLEG): container finished" podID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerID="14023920013a49e0e5b8695410d2770cf5d6264a21f66a9473567559748c4641" exitCode=0
Sep 16 19:24:59.920344 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:24:59.919924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" event={"ID":"53506d68-8ec9-48cc-9425-a127d9c8df28","Type":"ContainerDied","Data":"14023920013a49e0e5b8695410d2770cf5d6264a21f66a9473567559748c4641"}
Sep 16 19:25:00.925963 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:00.925928    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" event={"ID":"53506d68-8ec9-48cc-9425-a127d9c8df28","Type":"ContainerStarted","Data":"2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b"}
Sep 16 19:25:00.926348 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:00.925998    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" event={"ID":"53506d68-8ec9-48cc-9425-a127d9c8df28","Type":"ContainerStarted","Data":"6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994"}
Sep 16 19:25:00.926348 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:00.926209    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:25:00.948972 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:00.948897    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" podStartSLOduration=2.824157361 podStartE2EDuration="6.948881809s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.124724448s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:00.948462449 +0000 UTC m=+2384.762051307" watchObservedRunningTime="2026-09-16 19:25:00.948881809 +0000 UTC m=+2384.762470667"
Sep 16 19:25:01.930857 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:01.930822    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:25:07.941641 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:07.941604    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:25:12.836317 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:12.836277    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:25:19.627159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:19.627124    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"]
Sep 16 19:25:19.627590 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:19.627362    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl" podUID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerName="kserve-container" containerID="cri-o://fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10" gracePeriod=30
Sep 16 19:25:19.627590 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:19.627432    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl" podUID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerName="kube-rbac-proxy" containerID="cri-o://a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8" gracePeriod=30
Sep 16 19:25:20.002769 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.002726    2565 generic.go:356] "Generic (PLEG): container finished" podID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerID="a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8" exitCode=2
Sep 16 19:25:20.002952 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.002782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl" event={"ID":"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f","Type":"ContainerDied","Data":"a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8"}
Sep 16 19:25:20.771838 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.771811    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:25:20.860925 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.860835    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kube-rbac-proxy-sar-config\") pod \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") "
Sep 16 19:25:20.860925 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.860886    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kserve-provision-location\") pod \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") "
Sep 16 19:25:20.860925 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.860908    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-proxy-tls\") pod \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") "
Sep 16 19:25:20.861218 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.860953    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfx8j\" (UniqueName: \"kubernetes.io/projected/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kube-api-access-nfx8j\") pod \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") "
Sep 16 19:25:20.861218 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.861003    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-cabundle-cert\") pod \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\" (UID: \"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f\") "
Sep 16 19:25:20.861331 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.861284    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" (UID: "2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:20.861419 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.861393    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" (UID: "2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:20.861481 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.861457    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" (UID: "2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:20.863204 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.863179    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" (UID: "2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:20.863204 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.863187    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kube-api-access-nfx8j" (OuterVolumeSpecName: "kube-api-access-nfx8j") pod "2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" (UID: "2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f"). InnerVolumeSpecName "kube-api-access-nfx8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:20.961989 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.961944    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:25:20.961989 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.961980    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:25:20.961989 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.961992    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfx8j\" (UniqueName: \"kubernetes.io/projected/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kube-api-access-nfx8j\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:25:20.961989 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.962000    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:25:20.962310 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:20.962009    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:25:21.008257 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.008220    2565 generic.go:356] "Generic (PLEG): container finished" podID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerID="fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10" exitCode=0
Sep 16 19:25:21.008428 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.008282    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"
Sep 16 19:25:21.008428 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.008287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl" event={"ID":"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f","Type":"ContainerDied","Data":"fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10"}
Sep 16 19:25:21.008428 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.008350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl" event={"ID":"2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f","Type":"ContainerDied","Data":"63d443c98b1cf887b4fffea42835ae181dfb6314fdc1ce7585c0536c336e9727"}
Sep 16 19:25:21.008428 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.008388    2565 scope.go:117] "RemoveContainer" containerID="a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8"
Sep 16 19:25:21.016917 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.016897    2565 scope.go:117] "RemoveContainer" containerID="fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10"
Sep 16 19:25:21.024471 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.024446    2565 scope.go:117] "RemoveContainer" containerID="270a0ad583615b15f8f1abf978d2211b36e235addfa0ce26c9287684487a2b3f"
Sep 16 19:25:21.031780 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.031753    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"]
Sep 16 19:25:21.032046 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.032033    2565 scope.go:117] "RemoveContainer" containerID="a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8"
Sep 16 19:25:21.032365 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:25:21.032345    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8\": container with ID starting with a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8 not found: ID does not exist" containerID="a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8"
Sep 16 19:25:21.032414 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.032376    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8"} err="failed to get container status \"a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8\": rpc error: code = NotFound desc = could not find container \"a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8\": container with ID starting with a8fb6a0718963fbafac04c367c2102cca3eae056170eaf7a457d655102a598c8 not found: ID does not exist"
Sep 16 19:25:21.032414 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.032396    2565 scope.go:117] "RemoveContainer" containerID="fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10"
Sep 16 19:25:21.032661 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:25:21.032645    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10\": container with ID starting with fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10 not found: ID does not exist" containerID="fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10"
Sep 16 19:25:21.032702 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.032666    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10"} err="failed to get container status \"fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10\": rpc error: code = NotFound desc = could not find container \"fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10\": container with ID starting with fa34c8c499a23900ab9193eb9e5b051491a75cdec446dd6e71e41a1b5fed8f10 not found: ID does not exist"
Sep 16 19:25:21.032702 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.032680    2565 scope.go:117] "RemoveContainer" containerID="270a0ad583615b15f8f1abf978d2211b36e235addfa0ce26c9287684487a2b3f"
Sep 16 19:25:21.032903 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:25:21.032888    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270a0ad583615b15f8f1abf978d2211b36e235addfa0ce26c9287684487a2b3f\": container with ID starting with 270a0ad583615b15f8f1abf978d2211b36e235addfa0ce26c9287684487a2b3f not found: ID does not exist" containerID="270a0ad583615b15f8f1abf978d2211b36e235addfa0ce26c9287684487a2b3f"
Sep 16 19:25:21.032947 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.032912    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270a0ad583615b15f8f1abf978d2211b36e235addfa0ce26c9287684487a2b3f"} err="failed to get container status \"270a0ad583615b15f8f1abf978d2211b36e235addfa0ce26c9287684487a2b3f\": rpc error: code = NotFound desc = could not find container \"270a0ad583615b15f8f1abf978d2211b36e235addfa0ce26c9287684487a2b3f\": container with ID starting with 270a0ad583615b15f8f1abf978d2211b36e235addfa0ce26c9287684487a2b3f not found: ID does not exist"
Sep 16 19:25:21.036087 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:21.036058    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-brpcl"]
Sep 16 19:25:22.751784 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:22.751748    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" path="/var/lib/kubelet/pods/2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f/volumes"
Sep 16 19:25:38.245025 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:38.244966    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:25:47.944566 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:47.944474    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:25:50.081155 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.081120    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"]
Sep 16 19:25:50.081542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.081408    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerName="kserve-container"
Sep 16 19:25:50.081542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.081419    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerName="kserve-container"
Sep 16 19:25:50.081542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.081432    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerName="storage-initializer"
Sep 16 19:25:50.081542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.081438    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerName="storage-initializer"
Sep 16 19:25:50.081542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.081447    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerName="kube-rbac-proxy"
Sep 16 19:25:50.081542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.081453    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerName="kube-rbac-proxy"
Sep 16 19:25:50.081542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.081507    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerName="kserve-container"
Sep 16 19:25:50.081542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.081517    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bcc41d5-b3a2-43e8-9d93-ea3b3f87e64f" containerName="kube-rbac-proxy"
Sep 16 19:25:50.084760 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.084742    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.088614 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.088585    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 16 19:25:50.088765 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.088647    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:25:50.099916 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.099883    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"]
Sep 16 19:25:50.200587 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.200533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdcfc\" (UniqueName: \"kubernetes.io/projected/bca03f78-2c47-490e-9793-b12a90d55ca5-kube-api-access-cdcfc\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.200779 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.200602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca03f78-2c47-490e-9793-b12a90d55ca5-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.200779 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.200651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca03f78-2c47-490e-9793-b12a90d55ca5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.200779 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.200670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca03f78-2c47-490e-9793-b12a90d55ca5-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.200779 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.200752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca03f78-2c47-490e-9793-b12a90d55ca5-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.301922 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.301881    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca03f78-2c47-490e-9793-b12a90d55ca5-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.302135 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.301943    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdcfc\" (UniqueName: \"kubernetes.io/projected/bca03f78-2c47-490e-9793-b12a90d55ca5-kube-api-access-cdcfc\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.302135 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.301974    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca03f78-2c47-490e-9793-b12a90d55ca5-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.302135 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.302003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca03f78-2c47-490e-9793-b12a90d55ca5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.302135 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.302023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca03f78-2c47-490e-9793-b12a90d55ca5-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.302324 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:25:50.302204    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 16 19:25:50.302324 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:25:50.302297    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca03f78-2c47-490e-9793-b12a90d55ca5-proxy-tls podName:bca03f78-2c47-490e-9793-b12a90d55ca5 nodeName:}" failed. No retries permitted until 2026-09-16 19:25:50.80227697 +0000 UTC m=+2434.615865807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bca03f78-2c47-490e-9793-b12a90d55ca5-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq" (UID: "bca03f78-2c47-490e-9793-b12a90d55ca5") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 16 19:25:50.302521 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.302499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca03f78-2c47-490e-9793-b12a90d55ca5-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.302688 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.302670    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca03f78-2c47-490e-9793-b12a90d55ca5-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.302753 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.302733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca03f78-2c47-490e-9793-b12a90d55ca5-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.313162 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.313140    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdcfc\" (UniqueName: \"kubernetes.io/projected/bca03f78-2c47-490e-9793-b12a90d55ca5-kube-api-access-cdcfc\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.805027 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.804990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca03f78-2c47-490e-9793-b12a90d55ca5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.807604 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.807570    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca03f78-2c47-490e-9793-b12a90d55ca5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:50.995154 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:50.995088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:51.268457 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:51.268432    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"]
Sep 16 19:25:51.270927 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:25:51.270901    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca03f78_2c47_490e_9793_b12a90d55ca5.slice/crio-5cb67379b7e274abaf90887ea5dd7a9c4f72f15b07b650181085b6141e2597da WatchSource:0}: Error finding container 5cb67379b7e274abaf90887ea5dd7a9c4f72f15b07b650181085b6141e2597da: Status 404 returned error can't find the container with id 5cb67379b7e274abaf90887ea5dd7a9c4f72f15b07b650181085b6141e2597da
Sep 16 19:25:52.127614 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:52.127575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq" event={"ID":"bca03f78-2c47-490e-9793-b12a90d55ca5","Type":"ContainerStarted","Data":"5d63c21413b144c513ff08c5b99f90a9bd31f73379dbd561928b4940d003f485"}
Sep 16 19:25:52.127785 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:52.127638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq" event={"ID":"bca03f78-2c47-490e-9793-b12a90d55ca5","Type":"ContainerStarted","Data":"5cb67379b7e274abaf90887ea5dd7a9c4f72f15b07b650181085b6141e2597da"}
Sep 16 19:25:55.026614 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:55.026581    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"]
Sep 16 19:25:55.027023 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:55.026808    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="kserve-container" containerID="cri-o://6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994" gracePeriod=30
Sep 16 19:25:55.027023 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:55.026858    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="kube-rbac-proxy" containerID="cri-o://2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b" gracePeriod=30
Sep 16 19:25:55.141697 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:55.141666    2565 generic.go:356] "Generic (PLEG): container finished" podID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerID="5d63c21413b144c513ff08c5b99f90a9bd31f73379dbd561928b4940d003f485" exitCode=0
Sep 16 19:25:55.141828 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:55.141737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq" event={"ID":"bca03f78-2c47-490e-9793-b12a90d55ca5","Type":"ContainerDied","Data":"5d63c21413b144c513ff08c5b99f90a9bd31f73379dbd561928b4940d003f485"}
Sep 16 19:25:55.144084 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:55.144062    2565 generic.go:356] "Generic (PLEG): container finished" podID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerID="2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b" exitCode=2
Sep 16 19:25:55.144207 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:55.144096    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" event={"ID":"53506d68-8ec9-48cc-9425-a127d9c8df28","Type":"ContainerDied","Data":"2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b"}
Sep 16 19:25:56.150838 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:56.150800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq" event={"ID":"bca03f78-2c47-490e-9793-b12a90d55ca5","Type":"ContainerStarted","Data":"60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f"}
Sep 16 19:25:56.151375 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:56.150870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq" event={"ID":"bca03f78-2c47-490e-9793-b12a90d55ca5","Type":"ContainerStarted","Data":"2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6"}
Sep 16 19:25:56.151375 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:56.151064    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:56.173288 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:56.173224    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq" podStartSLOduration=2.402676715 podStartE2EDuration="6.173207547s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.770530832s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:56.172179583 +0000 UTC m=+2439.985768442" watchObservedRunningTime="2026-09-16 19:25:56.173207547 +0000 UTC m=+2439.986796406"
Sep 16 19:25:57.155882 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:57.155829    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:25:57.935846 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:57.935799    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 16 19:25:58.983357 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:25:58.983288    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:26:02.935333 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:02.935284    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 16 19:26:03.167482 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.167449    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:26:03.684764 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.684724    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:26:03.716203 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.716028    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53506d68-8ec9-48cc-9425-a127d9c8df28-cabundle-cert\") pod \"53506d68-8ec9-48cc-9425-a127d9c8df28\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") "
Sep 16 19:26:03.716203 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.716087    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53506d68-8ec9-48cc-9425-a127d9c8df28-proxy-tls\") pod \"53506d68-8ec9-48cc-9425-a127d9c8df28\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") "
Sep 16 19:26:03.716203 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.716166    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53506d68-8ec9-48cc-9425-a127d9c8df28-kube-rbac-proxy-sar-config\") pod \"53506d68-8ec9-48cc-9425-a127d9c8df28\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") "
Sep 16 19:26:03.716573 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.716219    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53506d68-8ec9-48cc-9425-a127d9c8df28-kserve-provision-location\") pod \"53506d68-8ec9-48cc-9425-a127d9c8df28\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") "
Sep 16 19:26:03.716573 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.716278    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhcrb\" (UniqueName: \"kubernetes.io/projected/53506d68-8ec9-48cc-9425-a127d9c8df28-kube-api-access-jhcrb\") pod \"53506d68-8ec9-48cc-9425-a127d9c8df28\" (UID: \"53506d68-8ec9-48cc-9425-a127d9c8df28\") "
Sep 16 19:26:03.716653 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.716599    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53506d68-8ec9-48cc-9425-a127d9c8df28-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53506d68-8ec9-48cc-9425-a127d9c8df28" (UID: "53506d68-8ec9-48cc-9425-a127d9c8df28"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:03.716653 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.716611    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53506d68-8ec9-48cc-9425-a127d9c8df28-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "53506d68-8ec9-48cc-9425-a127d9c8df28" (UID: "53506d68-8ec9-48cc-9425-a127d9c8df28"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:26:03.716653 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.716627    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53506d68-8ec9-48cc-9425-a127d9c8df28-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "53506d68-8ec9-48cc-9425-a127d9c8df28" (UID: "53506d68-8ec9-48cc-9425-a127d9c8df28"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:26:03.719009 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.718953    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53506d68-8ec9-48cc-9425-a127d9c8df28-kube-api-access-jhcrb" (OuterVolumeSpecName: "kube-api-access-jhcrb") pod "53506d68-8ec9-48cc-9425-a127d9c8df28" (UID: "53506d68-8ec9-48cc-9425-a127d9c8df28"). InnerVolumeSpecName "kube-api-access-jhcrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:03.719009 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.718953    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53506d68-8ec9-48cc-9425-a127d9c8df28-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "53506d68-8ec9-48cc-9425-a127d9c8df28" (UID: "53506d68-8ec9-48cc-9425-a127d9c8df28"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:03.817644 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.817589    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53506d68-8ec9-48cc-9425-a127d9c8df28-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:03.817644 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.817632    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53506d68-8ec9-48cc-9425-a127d9c8df28-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:03.817644 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.817647    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53506d68-8ec9-48cc-9425-a127d9c8df28-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:03.817926 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.817664    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53506d68-8ec9-48cc-9425-a127d9c8df28-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:03.817926 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:03.817677    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhcrb\" (UniqueName: \"kubernetes.io/projected/53506d68-8ec9-48cc-9425-a127d9c8df28-kube-api-access-jhcrb\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:04.189455 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.189368    2565 generic.go:356] "Generic (PLEG): container finished" podID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerID="6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994" exitCode=0
Sep 16 19:26:04.189455 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.189450    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"
Sep 16 19:26:04.189913 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.189451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" event={"ID":"53506d68-8ec9-48cc-9425-a127d9c8df28","Type":"ContainerDied","Data":"6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994"}
Sep 16 19:26:04.189913 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.189515    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq" event={"ID":"53506d68-8ec9-48cc-9425-a127d9c8df28","Type":"ContainerDied","Data":"fad2ed161a8c10023455f143d5ce5bb871c36bfc34846a7111d3c3f153041ef6"}
Sep 16 19:26:04.189913 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.189546    2565 scope.go:117] "RemoveContainer" containerID="2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b"
Sep 16 19:26:04.197986 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.197962    2565 scope.go:117] "RemoveContainer" containerID="6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994"
Sep 16 19:26:04.206262 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.206234    2565 scope.go:117] "RemoveContainer" containerID="14023920013a49e0e5b8695410d2770cf5d6264a21f66a9473567559748c4641"
Sep 16 19:26:04.213165 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.213140    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"]
Sep 16 19:26:04.214123 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.214087    2565 scope.go:117] "RemoveContainer" containerID="2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b"
Sep 16 19:26:04.214436 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:26:04.214417    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b\": container with ID starting with 2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b not found: ID does not exist" containerID="2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b"
Sep 16 19:26:04.214509 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.214444    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b"} err="failed to get container status \"2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b\": rpc error: code = NotFound desc = could not find container \"2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b\": container with ID starting with 2d0f5e6e789fd3e87b0ea79c4705ce46ce8a78d42ac283d7f7f247f49d6cfe2b not found: ID does not exist"
Sep 16 19:26:04.214509 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.214460    2565 scope.go:117] "RemoveContainer" containerID="6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994"
Sep 16 19:26:04.214701 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:26:04.214685    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994\": container with ID starting with 6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994 not found: ID does not exist" containerID="6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994"
Sep 16 19:26:04.214750 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.214706    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994"} err="failed to get container status \"6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994\": rpc error: code = NotFound desc = could not find container \"6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994\": container with ID starting with 6bcf1e6e8173335c0a1ed68c4d0aed6ad990eaca29ad2028375722a91a45b994 not found: ID does not exist"
Sep 16 19:26:04.214750 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.214720    2565 scope.go:117] "RemoveContainer" containerID="14023920013a49e0e5b8695410d2770cf5d6264a21f66a9473567559748c4641"
Sep 16 19:26:04.215177 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:26:04.215091    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14023920013a49e0e5b8695410d2770cf5d6264a21f66a9473567559748c4641\": container with ID starting with 14023920013a49e0e5b8695410d2770cf5d6264a21f66a9473567559748c4641 not found: ID does not exist" containerID="14023920013a49e0e5b8695410d2770cf5d6264a21f66a9473567559748c4641"
Sep 16 19:26:04.215304 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.215186    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14023920013a49e0e5b8695410d2770cf5d6264a21f66a9473567559748c4641"} err="failed to get container status \"14023920013a49e0e5b8695410d2770cf5d6264a21f66a9473567559748c4641\": rpc error: code = NotFound desc = could not find container \"14023920013a49e0e5b8695410d2770cf5d6264a21f66a9473567559748c4641\": container with ID starting with 14023920013a49e0e5b8695410d2770cf5d6264a21f66a9473567559748c4641 not found: ID does not exist"
Sep 16 19:26:04.217079 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.217057    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jwkzq"]
Sep 16 19:26:04.750742 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:04.750706    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" path="/var/lib/kubelet/pods/53506d68-8ec9-48cc-9425-a127d9c8df28/volumes"
Sep 16 19:26:33.172223 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:33.172186    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:26:40.158233 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:40.158193    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"]
Sep 16 19:26:40.158781 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:40.158445    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq" podUID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerName="kserve-container" containerID="cri-o://2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6" gracePeriod=30
Sep 16 19:26:40.158781 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:40.158501    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq" podUID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerName="kube-rbac-proxy" containerID="cri-o://60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f" gracePeriod=30
Sep 16 19:26:40.329340 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:40.329303    2565 generic.go:356] "Generic (PLEG): container finished" podID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerID="60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f" exitCode=2
Sep 16 19:26:40.329531 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:40.329380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq" event={"ID":"bca03f78-2c47-490e-9793-b12a90d55ca5","Type":"ContainerDied","Data":"60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f"}
Sep 16 19:26:41.708883 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.708856    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:26:41.831072 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.830955    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca03f78-2c47-490e-9793-b12a90d55ca5-kube-rbac-proxy-sar-config\") pod \"bca03f78-2c47-490e-9793-b12a90d55ca5\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") "
Sep 16 19:26:41.831072 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.831001    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdcfc\" (UniqueName: \"kubernetes.io/projected/bca03f78-2c47-490e-9793-b12a90d55ca5-kube-api-access-cdcfc\") pod \"bca03f78-2c47-490e-9793-b12a90d55ca5\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") "
Sep 16 19:26:41.831072 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.831039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca03f78-2c47-490e-9793-b12a90d55ca5-proxy-tls\") pod \"bca03f78-2c47-490e-9793-b12a90d55ca5\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") "
Sep 16 19:26:41.831072 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.831071    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca03f78-2c47-490e-9793-b12a90d55ca5-kserve-provision-location\") pod \"bca03f78-2c47-490e-9793-b12a90d55ca5\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") "
Sep 16 19:26:41.831522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.831129    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca03f78-2c47-490e-9793-b12a90d55ca5-cabundle-cert\") pod \"bca03f78-2c47-490e-9793-b12a90d55ca5\" (UID: \"bca03f78-2c47-490e-9793-b12a90d55ca5\") "
Sep 16 19:26:41.831522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.831499    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca03f78-2c47-490e-9793-b12a90d55ca5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bca03f78-2c47-490e-9793-b12a90d55ca5" (UID: "bca03f78-2c47-490e-9793-b12a90d55ca5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:26:41.831610 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.831543    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca03f78-2c47-490e-9793-b12a90d55ca5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bca03f78-2c47-490e-9793-b12a90d55ca5" (UID: "bca03f78-2c47-490e-9793-b12a90d55ca5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:41.831610 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.831587    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca03f78-2c47-490e-9793-b12a90d55ca5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bca03f78-2c47-490e-9793-b12a90d55ca5" (UID: "bca03f78-2c47-490e-9793-b12a90d55ca5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:26:41.833262 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.833235    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca03f78-2c47-490e-9793-b12a90d55ca5-kube-api-access-cdcfc" (OuterVolumeSpecName: "kube-api-access-cdcfc") pod "bca03f78-2c47-490e-9793-b12a90d55ca5" (UID: "bca03f78-2c47-490e-9793-b12a90d55ca5"). InnerVolumeSpecName "kube-api-access-cdcfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:41.833381 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.833294    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca03f78-2c47-490e-9793-b12a90d55ca5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bca03f78-2c47-490e-9793-b12a90d55ca5" (UID: "bca03f78-2c47-490e-9793-b12a90d55ca5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:41.931609 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.931570    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca03f78-2c47-490e-9793-b12a90d55ca5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:41.931609 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.931603    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdcfc\" (UniqueName: \"kubernetes.io/projected/bca03f78-2c47-490e-9793-b12a90d55ca5-kube-api-access-cdcfc\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:41.931609 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.931614    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca03f78-2c47-490e-9793-b12a90d55ca5-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:41.931609 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.931624    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca03f78-2c47-490e-9793-b12a90d55ca5-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:41.931886 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:41.931636    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca03f78-2c47-490e-9793-b12a90d55ca5-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:26:42.340918 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.340858    2565 generic.go:356] "Generic (PLEG): container finished" podID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerID="2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6" exitCode=0
Sep 16 19:26:42.341261 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.340943    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"
Sep 16 19:26:42.341261 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.340951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq" event={"ID":"bca03f78-2c47-490e-9793-b12a90d55ca5","Type":"ContainerDied","Data":"2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6"}
Sep 16 19:26:42.341261 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.341072    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq" event={"ID":"bca03f78-2c47-490e-9793-b12a90d55ca5","Type":"ContainerDied","Data":"5cb67379b7e274abaf90887ea5dd7a9c4f72f15b07b650181085b6141e2597da"}
Sep 16 19:26:42.341261 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.341144    2565 scope.go:117] "RemoveContainer" containerID="60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f"
Sep 16 19:26:42.352268 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.352244    2565 scope.go:117] "RemoveContainer" containerID="2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6"
Sep 16 19:26:42.361921 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.361886    2565 scope.go:117] "RemoveContainer" containerID="5d63c21413b144c513ff08c5b99f90a9bd31f73379dbd561928b4940d003f485"
Sep 16 19:26:42.366947 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.366919    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"]
Sep 16 19:26:42.375450 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.371591    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4gcmq"]
Sep 16 19:26:42.375756 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.375676    2565 scope.go:117] "RemoveContainer" containerID="60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f"
Sep 16 19:26:42.376444 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:26:42.376413    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f\": container with ID starting with 60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f not found: ID does not exist" containerID="60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f"
Sep 16 19:26:42.376558 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.376456    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f"} err="failed to get container status \"60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f\": rpc error: code = NotFound desc = could not find container \"60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f\": container with ID starting with 60b0adc60b72ed50313f91ed14769968912e2ee1a0e5dcea621bad3c4828355f not found: ID does not exist"
Sep 16 19:26:42.376558 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.376480    2565 scope.go:117] "RemoveContainer" containerID="2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6"
Sep 16 19:26:42.376836 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:26:42.376810    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6\": container with ID starting with 2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6 not found: ID does not exist" containerID="2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6"
Sep 16 19:26:42.376885 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.376842    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6"} err="failed to get container status \"2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6\": rpc error: code = NotFound desc = could not find container \"2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6\": container with ID starting with 2ed1f52ef12a50f7d62519ae979ec544a34941d87e151e4f7ac85b8f15c55bb6 not found: ID does not exist"
Sep 16 19:26:42.376885 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.376858    2565 scope.go:117] "RemoveContainer" containerID="5d63c21413b144c513ff08c5b99f90a9bd31f73379dbd561928b4940d003f485"
Sep 16 19:26:42.377151 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:26:42.377132    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d63c21413b144c513ff08c5b99f90a9bd31f73379dbd561928b4940d003f485\": container with ID starting with 5d63c21413b144c513ff08c5b99f90a9bd31f73379dbd561928b4940d003f485 not found: ID does not exist" containerID="5d63c21413b144c513ff08c5b99f90a9bd31f73379dbd561928b4940d003f485"
Sep 16 19:26:42.377191 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.377155    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d63c21413b144c513ff08c5b99f90a9bd31f73379dbd561928b4940d003f485"} err="failed to get container status \"5d63c21413b144c513ff08c5b99f90a9bd31f73379dbd561928b4940d003f485\": rpc error: code = NotFound desc = could not find container \"5d63c21413b144c513ff08c5b99f90a9bd31f73379dbd561928b4940d003f485\": container with ID starting with 5d63c21413b144c513ff08c5b99f90a9bd31f73379dbd561928b4940d003f485 not found: ID does not exist"
Sep 16 19:26:42.751119 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:42.751075    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bca03f78-2c47-490e-9793-b12a90d55ca5" path="/var/lib/kubelet/pods/bca03f78-2c47-490e-9793-b12a90d55ca5/volumes"
Sep 16 19:26:45.342094 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342058    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"]
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342359    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerName="kube-rbac-proxy"
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342372    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerName="kube-rbac-proxy"
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342388    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerName="storage-initializer"
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342394    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerName="storage-initializer"
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342400    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="kube-rbac-proxy"
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342406    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="kube-rbac-proxy"
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342416    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="storage-initializer"
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342425    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="storage-initializer"
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342431    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerName="kserve-container"
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342435    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerName="kserve-container"
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342441    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="kserve-container"
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342446    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="kserve-container"
Sep 16 19:26:45.342499 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342502    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="kube-rbac-proxy"
Sep 16 19:26:45.342906 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342510    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerName="kube-rbac-proxy"
Sep 16 19:26:45.342906 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342516    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="53506d68-8ec9-48cc-9425-a127d9c8df28" containerName="kserve-container"
Sep 16 19:26:45.342906 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.342521    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca03f78-2c47-490e-9793-b12a90d55ca5" containerName="kserve-container"
Sep 16 19:26:45.347329 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.347310    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.354420 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.354399    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 16 19:26:45.354503 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.354470    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 16 19:26:45.354824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.354803    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:26:45.355339 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.355318    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:26:45.355339 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.355335    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:26:45.355465 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.355398    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:26:45.355465 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.355429    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:26:45.361826 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.361803    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"]
Sep 16 19:26:45.461585 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.461553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e7a50fd-520b-4f63-9116-306946168928-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.461762 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.461592    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e7a50fd-520b-4f63-9116-306946168928-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.461762 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.461620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e7a50fd-520b-4f63-9116-306946168928-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.461762 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.461647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44gd\" (UniqueName: \"kubernetes.io/projected/3e7a50fd-520b-4f63-9116-306946168928-kube-api-access-p44gd\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.461762 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.461691    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7a50fd-520b-4f63-9116-306946168928-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.562673 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.562629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7a50fd-520b-4f63-9116-306946168928-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.562863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.562683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e7a50fd-520b-4f63-9116-306946168928-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.562863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.562709    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e7a50fd-520b-4f63-9116-306946168928-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.562863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.562726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e7a50fd-520b-4f63-9116-306946168928-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.562863 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.562743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44gd\" (UniqueName: \"kubernetes.io/projected/3e7a50fd-520b-4f63-9116-306946168928-kube-api-access-p44gd\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.563176 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.563154    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7a50fd-520b-4f63-9116-306946168928-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.563464 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.563441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e7a50fd-520b-4f63-9116-306946168928-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.563543 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.563512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e7a50fd-520b-4f63-9116-306946168928-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.565285 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.565263    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e7a50fd-520b-4f63-9116-306946168928-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.572001 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.571974    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44gd\" (UniqueName: \"kubernetes.io/projected/3e7a50fd-520b-4f63-9116-306946168928-kube-api-access-p44gd\") pod \"isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.658042 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.657950    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:45.936691 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:45.936660    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"]
Sep 16 19:26:45.939042 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:26:45.939010    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7a50fd_520b_4f63_9116_306946168928.slice/crio-0553f04698e212935daee55dc2490fdb5aa1b593b8edef82241735cb2b10ffa8 WatchSource:0}: Error finding container 0553f04698e212935daee55dc2490fdb5aa1b593b8edef82241735cb2b10ffa8: Status 404 returned error can't find the container with id 0553f04698e212935daee55dc2490fdb5aa1b593b8edef82241735cb2b10ffa8
Sep 16 19:26:46.358472 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:46.358434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" event={"ID":"3e7a50fd-520b-4f63-9116-306946168928","Type":"ContainerStarted","Data":"f46de403037dd871bcb4048dbc264cade7956d20f89e25ef9b11c86f8462dc7c"}
Sep 16 19:26:46.358885 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:46.358500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" event={"ID":"3e7a50fd-520b-4f63-9116-306946168928","Type":"ContainerStarted","Data":"0553f04698e212935daee55dc2490fdb5aa1b593b8edef82241735cb2b10ffa8"}
Sep 16 19:26:50.376982 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:50.376891    2565 generic.go:356] "Generic (PLEG): container finished" podID="3e7a50fd-520b-4f63-9116-306946168928" containerID="f46de403037dd871bcb4048dbc264cade7956d20f89e25ef9b11c86f8462dc7c" exitCode=0
Sep 16 19:26:50.376982 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:50.376964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" event={"ID":"3e7a50fd-520b-4f63-9116-306946168928","Type":"ContainerDied","Data":"f46de403037dd871bcb4048dbc264cade7956d20f89e25ef9b11c86f8462dc7c"}
Sep 16 19:26:50.435700 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:50.435675    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:26:51.385762 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:51.385711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" event={"ID":"3e7a50fd-520b-4f63-9116-306946168928","Type":"ContainerStarted","Data":"244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e"}
Sep 16 19:26:53.395995 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:53.395955    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" event={"ID":"3e7a50fd-520b-4f63-9116-306946168928","Type":"ContainerStarted","Data":"56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad"}
Sep 16 19:26:53.396511 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:53.396018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" event={"ID":"3e7a50fd-520b-4f63-9116-306946168928","Type":"ContainerStarted","Data":"9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992"}
Sep 16 19:26:53.396511 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:53.396221    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:53.421522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:53.421412    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podStartSLOduration=1.6156295 podStartE2EDuration="8.421392145s" totalImagesPullingTime="2.713148783s" totalInitContainerRuntime="4.092613862s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:53.419221861 +0000 UTC m=+2497.232810721" watchObservedRunningTime="2026-09-16 19:26:53.421392145 +0000 UTC m=+2497.234981006"
Sep 16 19:26:54.400392 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:54.400359    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:26:54.400839 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:26:54.400660    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:27:00.409857 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:27:00.409824    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:27:20.411098 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:27:20.411051    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:27:25.658665 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:27:25.658612    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:27:30.414922 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:27:30.414889    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:28:00.416643 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:00.416613    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:28:05.397802 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:05.397767    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"]
Sep 16 19:28:05.398250 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:05.398034    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-container" containerID="cri-o://244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e" gracePeriod=30
Sep 16 19:28:05.398250 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:05.398118    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kube-rbac-proxy" containerID="cri-o://56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad" gracePeriod=30
Sep 16 19:28:05.398250 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:05.398156    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-agent" containerID="cri-o://9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992" gracePeriod=30
Sep 16 19:28:05.404685 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:05.404657    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 16 19:28:05.670566 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:05.670480    2565 generic.go:356] "Generic (PLEG): container finished" podID="3e7a50fd-520b-4f63-9116-306946168928" containerID="56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad" exitCode=2
Sep 16 19:28:05.670566 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:05.670548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" event={"ID":"3e7a50fd-520b-4f63-9116-306946168928","Type":"ContainerDied","Data":"56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad"}
Sep 16 19:28:07.681003 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:07.680974    2565 generic.go:356] "Generic (PLEG): container finished" podID="3e7a50fd-520b-4f63-9116-306946168928" containerID="244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e" exitCode=0
Sep 16 19:28:07.681367 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:07.681033    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" event={"ID":"3e7a50fd-520b-4f63-9116-306946168928","Type":"ContainerDied","Data":"244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e"}
Sep 16 19:28:10.404772 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:10.404728    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 16 19:28:10.411348 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:10.411298    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:28:15.404879 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:15.404825    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 16 19:28:15.405321 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:15.404970    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:28:20.404453 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:20.404401    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 16 19:28:20.411063 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:20.411021    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:28:25.405445 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:25.405376    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 16 19:28:30.405230 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:30.405178    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 16 19:28:30.410709 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:30.410672    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:28:30.410827 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:30.410815    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:28:35.404746 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.404701    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 16 19:28:35.543021 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.542989    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:28:35.622489 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.622447    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e7a50fd-520b-4f63-9116-306946168928-kube-rbac-proxy-sar-config\") pod \"3e7a50fd-520b-4f63-9116-306946168928\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") "
Sep 16 19:28:35.622679 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.622507    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e7a50fd-520b-4f63-9116-306946168928-cabundle-cert\") pod \"3e7a50fd-520b-4f63-9116-306946168928\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") "
Sep 16 19:28:35.622679 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.622527    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p44gd\" (UniqueName: \"kubernetes.io/projected/3e7a50fd-520b-4f63-9116-306946168928-kube-api-access-p44gd\") pod \"3e7a50fd-520b-4f63-9116-306946168928\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") "
Sep 16 19:28:35.622679 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.622555    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e7a50fd-520b-4f63-9116-306946168928-proxy-tls\") pod \"3e7a50fd-520b-4f63-9116-306946168928\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") "
Sep 16 19:28:35.622679 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.622582    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7a50fd-520b-4f63-9116-306946168928-kserve-provision-location\") pod \"3e7a50fd-520b-4f63-9116-306946168928\" (UID: \"3e7a50fd-520b-4f63-9116-306946168928\") "
Sep 16 19:28:35.622967 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.622938    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7a50fd-520b-4f63-9116-306946168928-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e7a50fd-520b-4f63-9116-306946168928" (UID: "3e7a50fd-520b-4f63-9116-306946168928"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:35.623023 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.622969    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7a50fd-520b-4f63-9116-306946168928-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e7a50fd-520b-4f63-9116-306946168928" (UID: "3e7a50fd-520b-4f63-9116-306946168928"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:35.623023 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.622951    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7a50fd-520b-4f63-9116-306946168928-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e7a50fd-520b-4f63-9116-306946168928" (UID: "3e7a50fd-520b-4f63-9116-306946168928"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:35.624906 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.624876    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e7a50fd-520b-4f63-9116-306946168928-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e7a50fd-520b-4f63-9116-306946168928" (UID: "3e7a50fd-520b-4f63-9116-306946168928"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:35.625004 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.624982    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7a50fd-520b-4f63-9116-306946168928-kube-api-access-p44gd" (OuterVolumeSpecName: "kube-api-access-p44gd") pod "3e7a50fd-520b-4f63-9116-306946168928" (UID: "3e7a50fd-520b-4f63-9116-306946168928"). InnerVolumeSpecName "kube-api-access-p44gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:35.723531 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.723493    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e7a50fd-520b-4f63-9116-306946168928-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:28:35.723531 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.723527    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e7a50fd-520b-4f63-9116-306946168928-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:28:35.723531 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.723538    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p44gd\" (UniqueName: \"kubernetes.io/projected/3e7a50fd-520b-4f63-9116-306946168928-kube-api-access-p44gd\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:28:35.723757 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.723549    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e7a50fd-520b-4f63-9116-306946168928-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:28:35.723757 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.723560    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7a50fd-520b-4f63-9116-306946168928-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:28:35.791184 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.791141    2565 generic.go:356] "Generic (PLEG): container finished" podID="3e7a50fd-520b-4f63-9116-306946168928" containerID="9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992" exitCode=0
Sep 16 19:28:35.791355 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.791223    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"
Sep 16 19:28:35.791355 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.791225    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" event={"ID":"3e7a50fd-520b-4f63-9116-306946168928","Type":"ContainerDied","Data":"9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992"}
Sep 16 19:28:35.791355 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.791295    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8" event={"ID":"3e7a50fd-520b-4f63-9116-306946168928","Type":"ContainerDied","Data":"0553f04698e212935daee55dc2490fdb5aa1b593b8edef82241735cb2b10ffa8"}
Sep 16 19:28:35.791355 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.791325    2565 scope.go:117] "RemoveContainer" containerID="56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad"
Sep 16 19:28:35.799952 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.799927    2565 scope.go:117] "RemoveContainer" containerID="9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992"
Sep 16 19:28:35.808361 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.808336    2565 scope.go:117] "RemoveContainer" containerID="244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e"
Sep 16 19:28:35.815561 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.815521    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"]
Sep 16 19:28:35.816670 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.816649    2565 scope.go:117] "RemoveContainer" containerID="f46de403037dd871bcb4048dbc264cade7956d20f89e25ef9b11c86f8462dc7c"
Sep 16 19:28:35.821981 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.821948    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6cdb4c58cd-487v8"]
Sep 16 19:28:35.825076 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.825057    2565 scope.go:117] "RemoveContainer" containerID="56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad"
Sep 16 19:28:35.825457 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:28:35.825434    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad\": container with ID starting with 56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad not found: ID does not exist" containerID="56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad"
Sep 16 19:28:35.825545 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.825467    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad"} err="failed to get container status \"56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad\": rpc error: code = NotFound desc = could not find container \"56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad\": container with ID starting with 56fa845cb3cd17c0d0afbdddd9aba396df03e3cb4344c3c08aa0dee4bfb9e5ad not found: ID does not exist"
Sep 16 19:28:35.825545 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.825489    2565 scope.go:117] "RemoveContainer" containerID="9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992"
Sep 16 19:28:35.825788 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:28:35.825772    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992\": container with ID starting with 9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992 not found: ID does not exist" containerID="9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992"
Sep 16 19:28:35.825830 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.825795    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992"} err="failed to get container status \"9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992\": rpc error: code = NotFound desc = could not find container \"9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992\": container with ID starting with 9d58fe15b2d2b722ccd4290c7adcfc71e249130473586d579bbb254e26ed5992 not found: ID does not exist"
Sep 16 19:28:35.825830 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.825811    2565 scope.go:117] "RemoveContainer" containerID="244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e"
Sep 16 19:28:35.826085 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:28:35.826068    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e\": container with ID starting with 244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e not found: ID does not exist" containerID="244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e"
Sep 16 19:28:35.826160 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.826086    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e"} err="failed to get container status \"244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e\": rpc error: code = NotFound desc = could not find container \"244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e\": container with ID starting with 244142741ea0de64deafe577f4a9b2810ca63e88f19860dd55be0a179ed84c5e not found: ID does not exist"
Sep 16 19:28:35.826160 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.826098    2565 scope.go:117] "RemoveContainer" containerID="f46de403037dd871bcb4048dbc264cade7956d20f89e25ef9b11c86f8462dc7c"
Sep 16 19:28:35.826406 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:28:35.826391    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46de403037dd871bcb4048dbc264cade7956d20f89e25ef9b11c86f8462dc7c\": container with ID starting with f46de403037dd871bcb4048dbc264cade7956d20f89e25ef9b11c86f8462dc7c not found: ID does not exist" containerID="f46de403037dd871bcb4048dbc264cade7956d20f89e25ef9b11c86f8462dc7c"
Sep 16 19:28:35.826449 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:35.826410    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46de403037dd871bcb4048dbc264cade7956d20f89e25ef9b11c86f8462dc7c"} err="failed to get container status \"f46de403037dd871bcb4048dbc264cade7956d20f89e25ef9b11c86f8462dc7c\": rpc error: code = NotFound desc = could not find container \"f46de403037dd871bcb4048dbc264cade7956d20f89e25ef9b11c86f8462dc7c\": container with ID starting with f46de403037dd871bcb4048dbc264cade7956d20f89e25ef9b11c86f8462dc7c not found: ID does not exist"
Sep 16 19:28:36.751276 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:28:36.751238    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e7a50fd-520b-4f63-9116-306946168928" path="/var/lib/kubelet/pods/3e7a50fd-520b-4f63-9116-306946168928/volumes"
Sep 16 19:29:19.831461 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.831417    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"]
Sep 16 19:29:19.831889 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.831687    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="storage-initializer"
Sep 16 19:29:19.831889 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.831702    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="storage-initializer"
Sep 16 19:29:19.831889 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.831715    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-agent"
Sep 16 19:29:19.831889 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.831722    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-agent"
Sep 16 19:29:19.831889 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.831730    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kube-rbac-proxy"
Sep 16 19:29:19.831889 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.831736    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kube-rbac-proxy"
Sep 16 19:29:19.831889 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.831745    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-container"
Sep 16 19:29:19.831889 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.831752    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-container"
Sep 16 19:29:19.831889 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.831806    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-container"
Sep 16 19:29:19.831889 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.831813    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kube-rbac-proxy"
Sep 16 19:29:19.831889 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.831820    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7a50fd-520b-4f63-9116-306946168928" containerName="kserve-agent"
Sep 16 19:29:19.835626 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.835608    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:19.838493 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.838467    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 16 19:29:19.838493 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.838473    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:29:19.838716 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.838499    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:29:19.839632 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.839601    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:29:19.839632 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.839620    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:29:19.839816 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.839626    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:29:19.839816 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.839648    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:29:19.850426 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.850400    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"]
Sep 16 19:29:19.964149 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.964090    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62b9056-3fcf-401f-bd88-99157c2f054b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:19.964149 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.964153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a62b9056-3fcf-401f-bd88-99157c2f054b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:19.964419 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.964199    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a62b9056-3fcf-401f-bd88-99157c2f054b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:19.964419 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.964233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a62b9056-3fcf-401f-bd88-99157c2f054b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:19.964419 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:19.964309    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8km5\" (UniqueName: \"kubernetes.io/projected/a62b9056-3fcf-401f-bd88-99157c2f054b-kube-api-access-t8km5\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:20.065226 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.065166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a62b9056-3fcf-401f-bd88-99157c2f054b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:20.065226 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.065237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a62b9056-3fcf-401f-bd88-99157c2f054b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:20.065575 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.065293    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8km5\" (UniqueName: \"kubernetes.io/projected/a62b9056-3fcf-401f-bd88-99157c2f054b-kube-api-access-t8km5\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:20.065575 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.065325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62b9056-3fcf-401f-bd88-99157c2f054b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:20.065575 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.065349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a62b9056-3fcf-401f-bd88-99157c2f054b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:20.065931 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.065900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62b9056-3fcf-401f-bd88-99157c2f054b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:20.066211 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.066187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a62b9056-3fcf-401f-bd88-99157c2f054b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:20.066211 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.066200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a62b9056-3fcf-401f-bd88-99157c2f054b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:20.068397 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.068369    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a62b9056-3fcf-401f-bd88-99157c2f054b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:20.092488 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.092421    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8km5\" (UniqueName: \"kubernetes.io/projected/a62b9056-3fcf-401f-bd88-99157c2f054b-kube-api-access-t8km5\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:20.145936 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.145900    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:20.420474 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.420393    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"]
Sep 16 19:29:20.423854 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:29:20.423820    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62b9056_3fcf_401f_bd88_99157c2f054b.slice/crio-48f06dec565ed3e3c7027c78c015a72c31b2a87e818977a45c9a024c4fb7f07e WatchSource:0}: Error finding container 48f06dec565ed3e3c7027c78c015a72c31b2a87e818977a45c9a024c4fb7f07e: Status 404 returned error can't find the container with id 48f06dec565ed3e3c7027c78c015a72c31b2a87e818977a45c9a024c4fb7f07e
Sep 16 19:29:20.948426 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.948386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" event={"ID":"a62b9056-3fcf-401f-bd88-99157c2f054b","Type":"ContainerStarted","Data":"ac707deeb874098a45ea23e1ba021d6bf72a2ec7c9e7d56af81e6a58ca653563"}
Sep 16 19:29:20.948922 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:20.948456    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" event={"ID":"a62b9056-3fcf-401f-bd88-99157c2f054b","Type":"ContainerStarted","Data":"48f06dec565ed3e3c7027c78c015a72c31b2a87e818977a45c9a024c4fb7f07e"}
Sep 16 19:29:21.954549 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:21.954510    2565 generic.go:356] "Generic (PLEG): container finished" podID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerID="ac707deeb874098a45ea23e1ba021d6bf72a2ec7c9e7d56af81e6a58ca653563" exitCode=0
Sep 16 19:29:21.955018 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:21.954598    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" event={"ID":"a62b9056-3fcf-401f-bd88-99157c2f054b","Type":"ContainerDied","Data":"ac707deeb874098a45ea23e1ba021d6bf72a2ec7c9e7d56af81e6a58ca653563"}
Sep 16 19:29:22.960059 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:22.960023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" event={"ID":"a62b9056-3fcf-401f-bd88-99157c2f054b","Type":"ContainerStarted","Data":"bf9941e8acb6944139f3ba363d6dab102a8b732bfcd98f690a00d0f0f026f54d"}
Sep 16 19:29:22.960492 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:22.960093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" event={"ID":"a62b9056-3fcf-401f-bd88-99157c2f054b","Type":"ContainerStarted","Data":"7b9a57f7db61229ada2f04cf985937568a460137ea8f6797b678acd6f8933e6a"}
Sep 16 19:29:22.960492 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:22.960150    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:22.960492 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:22.960257    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:22.961579 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:22.961551    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:29:22.982780 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:22.982718    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podStartSLOduration=2.595942 podStartE2EDuration="3.982698635s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.386756635s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:22.981892267 +0000 UTC m=+2646.795481128" watchObservedRunningTime="2026-09-16 19:29:22.982698635 +0000 UTC m=+2646.796287546"
Sep 16 19:29:23.964358 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:23.964319    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:29:28.969913 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:28.969882    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:29:28.970582 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:28.970554    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:29:38.970614 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:38.970572    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:29:48.971008 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:48.970962    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:29:58.970608 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:29:58.970567    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:30:08.971289 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:08.971242    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:30:18.971373 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:18.971281    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:30:28.971213 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:28.971167    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:30:38.971152 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:38.971084    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:30:48.972477 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:48.972445    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:30:49.834020 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:49.833986    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"]
Sep 16 19:30:49.834280 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:49.834235    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container" containerID="cri-o://7b9a57f7db61229ada2f04cf985937568a460137ea8f6797b678acd6f8933e6a" gracePeriod=30
Sep 16 19:30:49.834356 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:49.834301    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kube-rbac-proxy" containerID="cri-o://bf9941e8acb6944139f3ba363d6dab102a8b732bfcd98f690a00d0f0f026f54d" gracePeriod=30
Sep 16 19:30:50.286319 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:50.286276    2565 generic.go:356] "Generic (PLEG): container finished" podID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerID="bf9941e8acb6944139f3ba363d6dab102a8b732bfcd98f690a00d0f0f026f54d" exitCode=2
Sep 16 19:30:50.286724 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:50.286353    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" event={"ID":"a62b9056-3fcf-401f-bd88-99157c2f054b","Type":"ContainerDied","Data":"bf9941e8acb6944139f3ba363d6dab102a8b732bfcd98f690a00d0f0f026f54d"}
Sep 16 19:30:50.969257 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:50.969220    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"]
Sep 16 19:30:50.972509 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:50.972486    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:50.975323 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:50.975295    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 16 19:30:50.975454 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:50.975322    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:30:50.984288 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:50.984263    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"]
Sep 16 19:30:51.043487 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.043450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs4s5\" (UniqueName: \"kubernetes.io/projected/5ca20bca-2880-440f-8e99-4ab964ef0543-kube-api-access-cs4s5\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.043487 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.043492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ca20bca-2880-440f-8e99-4ab964ef0543-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.043723 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.043598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ca20bca-2880-440f-8e99-4ab964ef0543-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.043723 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.043641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ca20bca-2880-440f-8e99-4ab964ef0543-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.144624 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.144578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ca20bca-2880-440f-8e99-4ab964ef0543-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.144810 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.144677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ca20bca-2880-440f-8e99-4ab964ef0543-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.144810 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.144725    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ca20bca-2880-440f-8e99-4ab964ef0543-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.144810 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.144781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs4s5\" (UniqueName: \"kubernetes.io/projected/5ca20bca-2880-440f-8e99-4ab964ef0543-kube-api-access-cs4s5\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.145148 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.145088    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ca20bca-2880-440f-8e99-4ab964ef0543-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.145417 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.145395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ca20bca-2880-440f-8e99-4ab964ef0543-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.147405 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.147379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ca20bca-2880-440f-8e99-4ab964ef0543-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.154309 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.154280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs4s5\" (UniqueName: \"kubernetes.io/projected/5ca20bca-2880-440f-8e99-4ab964ef0543-kube-api-access-cs4s5\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.282702 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.282594    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:30:51.560175 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:51.560148    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"]
Sep 16 19:30:51.562593 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:30:51.562560    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca20bca_2880_440f_8e99_4ab964ef0543.slice/crio-c079c0faf93ecce647cd9aa5ad57a2a37936b5cde9f48da1ca73bdbcee862e3e WatchSource:0}: Error finding container c079c0faf93ecce647cd9aa5ad57a2a37936b5cde9f48da1ca73bdbcee862e3e: Status 404 returned error can't find the container with id c079c0faf93ecce647cd9aa5ad57a2a37936b5cde9f48da1ca73bdbcee862e3e
Sep 16 19:30:52.295865 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:52.295825    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq" event={"ID":"5ca20bca-2880-440f-8e99-4ab964ef0543","Type":"ContainerStarted","Data":"7ac64b6532ef01d69204153178ab53b2637c87043d4059cda2afbb2bdb1ee928"}
Sep 16 19:30:52.296059 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:52.295889    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq" event={"ID":"5ca20bca-2880-440f-8e99-4ab964ef0543","Type":"ContainerStarted","Data":"c079c0faf93ecce647cd9aa5ad57a2a37936b5cde9f48da1ca73bdbcee862e3e"}
Sep 16 19:30:53.964671 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:53.964626    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 16 19:30:55.315450 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.315414    2565 generic.go:356] "Generic (PLEG): container finished" podID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerID="7b9a57f7db61229ada2f04cf985937568a460137ea8f6797b678acd6f8933e6a" exitCode=0
Sep 16 19:30:55.315899 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.315492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" event={"ID":"a62b9056-3fcf-401f-bd88-99157c2f054b","Type":"ContainerDied","Data":"7b9a57f7db61229ada2f04cf985937568a460137ea8f6797b678acd6f8933e6a"}
Sep 16 19:30:55.318132 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.318092    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq_5ca20bca-2880-440f-8e99-4ab964ef0543/storage-initializer/0.log"
Sep 16 19:30:55.318300 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.318146    2565 generic.go:356] "Generic (PLEG): container finished" podID="5ca20bca-2880-440f-8e99-4ab964ef0543" containerID="7ac64b6532ef01d69204153178ab53b2637c87043d4059cda2afbb2bdb1ee928" exitCode=1
Sep 16 19:30:55.318300 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.318183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq" event={"ID":"5ca20bca-2880-440f-8e99-4ab964ef0543","Type":"ContainerDied","Data":"7ac64b6532ef01d69204153178ab53b2637c87043d4059cda2afbb2bdb1ee928"}
Sep 16 19:30:55.385702 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.385673    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:30:55.480971 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.480926    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a62b9056-3fcf-401f-bd88-99157c2f054b-kube-rbac-proxy-sar-config\") pod \"a62b9056-3fcf-401f-bd88-99157c2f054b\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") "
Sep 16 19:30:55.481206 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.480980    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62b9056-3fcf-401f-bd88-99157c2f054b-kserve-provision-location\") pod \"a62b9056-3fcf-401f-bd88-99157c2f054b\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") "
Sep 16 19:30:55.481206 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.481030    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a62b9056-3fcf-401f-bd88-99157c2f054b-proxy-tls\") pod \"a62b9056-3fcf-401f-bd88-99157c2f054b\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") "
Sep 16 19:30:55.481206 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.481088    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a62b9056-3fcf-401f-bd88-99157c2f054b-cabundle-cert\") pod \"a62b9056-3fcf-401f-bd88-99157c2f054b\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") "
Sep 16 19:30:55.481206 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.481137    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8km5\" (UniqueName: \"kubernetes.io/projected/a62b9056-3fcf-401f-bd88-99157c2f054b-kube-api-access-t8km5\") pod \"a62b9056-3fcf-401f-bd88-99157c2f054b\" (UID: \"a62b9056-3fcf-401f-bd88-99157c2f054b\") "
Sep 16 19:30:55.481678 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.481414    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62b9056-3fcf-401f-bd88-99157c2f054b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a62b9056-3fcf-401f-bd88-99157c2f054b" (UID: "a62b9056-3fcf-401f-bd88-99157c2f054b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:55.481678 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.481440    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62b9056-3fcf-401f-bd88-99157c2f054b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a62b9056-3fcf-401f-bd88-99157c2f054b" (UID: "a62b9056-3fcf-401f-bd88-99157c2f054b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:55.481678 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.481470    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62b9056-3fcf-401f-bd88-99157c2f054b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a62b9056-3fcf-401f-bd88-99157c2f054b" (UID: "a62b9056-3fcf-401f-bd88-99157c2f054b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:55.483466 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.483440    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62b9056-3fcf-401f-bd88-99157c2f054b-kube-api-access-t8km5" (OuterVolumeSpecName: "kube-api-access-t8km5") pod "a62b9056-3fcf-401f-bd88-99157c2f054b" (UID: "a62b9056-3fcf-401f-bd88-99157c2f054b"). InnerVolumeSpecName "kube-api-access-t8km5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:55.483530 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.483486    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62b9056-3fcf-401f-bd88-99157c2f054b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a62b9056-3fcf-401f-bd88-99157c2f054b" (UID: "a62b9056-3fcf-401f-bd88-99157c2f054b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:55.582686 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.582641    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a62b9056-3fcf-401f-bd88-99157c2f054b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:30:55.582686 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.582688    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62b9056-3fcf-401f-bd88-99157c2f054b-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:30:55.582892 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.582703    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a62b9056-3fcf-401f-bd88-99157c2f054b-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:30:55.582892 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.582719    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a62b9056-3fcf-401f-bd88-99157c2f054b-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:30:55.582892 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:55.582733    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8km5\" (UniqueName: \"kubernetes.io/projected/a62b9056-3fcf-401f-bd88-99157c2f054b-kube-api-access-t8km5\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:30:56.323616 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:56.323587    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq_5ca20bca-2880-440f-8e99-4ab964ef0543/storage-initializer/0.log"
Sep 16 19:30:56.324045 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:56.323703    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq" event={"ID":"5ca20bca-2880-440f-8e99-4ab964ef0543","Type":"ContainerStarted","Data":"7e0e453ab89d6d61fff9c61ed7085a5185392c039fd15672899dde84efe8810b"}
Sep 16 19:30:56.325602 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:56.325577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6" event={"ID":"a62b9056-3fcf-401f-bd88-99157c2f054b","Type":"ContainerDied","Data":"48f06dec565ed3e3c7027c78c015a72c31b2a87e818977a45c9a024c4fb7f07e"}
Sep 16 19:30:56.325745 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:56.325624    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"
Sep 16 19:30:56.325745 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:56.325635    2565 scope.go:117] "RemoveContainer" containerID="bf9941e8acb6944139f3ba363d6dab102a8b732bfcd98f690a00d0f0f026f54d"
Sep 16 19:30:56.333722 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:56.333706    2565 scope.go:117] "RemoveContainer" containerID="7b9a57f7db61229ada2f04cf985937568a460137ea8f6797b678acd6f8933e6a"
Sep 16 19:30:56.340753 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:56.340736    2565 scope.go:117] "RemoveContainer" containerID="ac707deeb874098a45ea23e1ba021d6bf72a2ec7c9e7d56af81e6a58ca653563"
Sep 16 19:30:56.359975 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:56.359944    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"]
Sep 16 19:30:56.363819 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:56.363792    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5b87bbf54c-sv9j6"]
Sep 16 19:30:56.752046 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:56.752009    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" path="/var/lib/kubelet/pods/a62b9056-3fcf-401f-bd88-99157c2f054b/volumes"
Sep 16 19:30:59.339867 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:59.339837    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq_5ca20bca-2880-440f-8e99-4ab964ef0543/storage-initializer/1.log"
Sep 16 19:30:59.340323 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:59.340211    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq_5ca20bca-2880-440f-8e99-4ab964ef0543/storage-initializer/0.log"
Sep 16 19:30:59.340323 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:59.340241    2565 generic.go:356] "Generic (PLEG): container finished" podID="5ca20bca-2880-440f-8e99-4ab964ef0543" containerID="7e0e453ab89d6d61fff9c61ed7085a5185392c039fd15672899dde84efe8810b" exitCode=1
Sep 16 19:30:59.340323 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:59.340288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq" event={"ID":"5ca20bca-2880-440f-8e99-4ab964ef0543","Type":"ContainerDied","Data":"7e0e453ab89d6d61fff9c61ed7085a5185392c039fd15672899dde84efe8810b"}
Sep 16 19:30:59.340426 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:59.340373    2565 scope.go:117] "RemoveContainer" containerID="7ac64b6532ef01d69204153178ab53b2637c87043d4059cda2afbb2bdb1ee928"
Sep 16 19:30:59.340755 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:59.340728    2565 scope.go:117] "RemoveContainer" containerID="7ac64b6532ef01d69204153178ab53b2637c87043d4059cda2afbb2bdb1ee928"
Sep 16 19:30:59.350871 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:30:59.350842    2565 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq_kserve-ci-e2e-test_5ca20bca-2880-440f-8e99-4ab964ef0543_0 in pod sandbox c079c0faf93ecce647cd9aa5ad57a2a37936b5cde9f48da1ca73bdbcee862e3e from index: no such id: '7ac64b6532ef01d69204153178ab53b2637c87043d4059cda2afbb2bdb1ee928'" containerID="7ac64b6532ef01d69204153178ab53b2637c87043d4059cda2afbb2bdb1ee928"
Sep 16 19:30:59.350944 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:30:59.350880    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq_kserve-ci-e2e-test(5ca20bca-2880-440f-8e99-4ab964ef0543)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq" podUID="5ca20bca-2880-440f-8e99-4ab964ef0543"
Sep 16 19:30:59.350944 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:30:59.350881    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac64b6532ef01d69204153178ab53b2637c87043d4059cda2afbb2bdb1ee928"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq_kserve-ci-e2e-test_5ca20bca-2880-440f-8e99-4ab964ef0543_0 in pod sandbox c079c0faf93ecce647cd9aa5ad57a2a37936b5cde9f48da1ca73bdbcee862e3e from index: no such id: '7ac64b6532ef01d69204153178ab53b2637c87043d4059cda2afbb2bdb1ee928'"
Sep 16 19:31:00.345475 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:00.345444    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq_5ca20bca-2880-440f-8e99-4ab964ef0543/storage-initializer/1.log"
Sep 16 19:31:00.953796 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:00.953748    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"]
Sep 16 19:31:01.080606 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.080582    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq_5ca20bca-2880-440f-8e99-4ab964ef0543/storage-initializer/1.log"
Sep 16 19:31:01.084037 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.084020    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:31:01.231957 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.231844    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs4s5\" (UniqueName: \"kubernetes.io/projected/5ca20bca-2880-440f-8e99-4ab964ef0543-kube-api-access-cs4s5\") pod \"5ca20bca-2880-440f-8e99-4ab964ef0543\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") "
Sep 16 19:31:01.231957 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.231910    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ca20bca-2880-440f-8e99-4ab964ef0543-kube-rbac-proxy-sar-config\") pod \"5ca20bca-2880-440f-8e99-4ab964ef0543\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") "
Sep 16 19:31:01.232240 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.231981    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ca20bca-2880-440f-8e99-4ab964ef0543-kserve-provision-location\") pod \"5ca20bca-2880-440f-8e99-4ab964ef0543\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") "
Sep 16 19:31:01.232240 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.232011    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ca20bca-2880-440f-8e99-4ab964ef0543-proxy-tls\") pod \"5ca20bca-2880-440f-8e99-4ab964ef0543\" (UID: \"5ca20bca-2880-440f-8e99-4ab964ef0543\") "
Sep 16 19:31:01.232340 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.232284    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca20bca-2880-440f-8e99-4ab964ef0543-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ca20bca-2880-440f-8e99-4ab964ef0543" (UID: "5ca20bca-2880-440f-8e99-4ab964ef0543"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:01.232386 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.232339    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca20bca-2880-440f-8e99-4ab964ef0543-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ca20bca-2880-440f-8e99-4ab964ef0543" (UID: "5ca20bca-2880-440f-8e99-4ab964ef0543"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:01.234149 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.234128    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca20bca-2880-440f-8e99-4ab964ef0543-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ca20bca-2880-440f-8e99-4ab964ef0543" (UID: "5ca20bca-2880-440f-8e99-4ab964ef0543"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:01.234250 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.234235    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca20bca-2880-440f-8e99-4ab964ef0543-kube-api-access-cs4s5" (OuterVolumeSpecName: "kube-api-access-cs4s5") pod "5ca20bca-2880-440f-8e99-4ab964ef0543" (UID: "5ca20bca-2880-440f-8e99-4ab964ef0543"). InnerVolumeSpecName "kube-api-access-cs4s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:01.333347 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.333305    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ca20bca-2880-440f-8e99-4ab964ef0543-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:31:01.333347 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.333340    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ca20bca-2880-440f-8e99-4ab964ef0543-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:31:01.333347 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.333354    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs4s5\" (UniqueName: \"kubernetes.io/projected/5ca20bca-2880-440f-8e99-4ab964ef0543-kube-api-access-cs4s5\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:31:01.333600 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.333366    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ca20bca-2880-440f-8e99-4ab964ef0543-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:31:01.351319 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.351291    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq_5ca20bca-2880-440f-8e99-4ab964ef0543/storage-initializer/1.log"
Sep 16 19:31:01.351689 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.351382    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq" event={"ID":"5ca20bca-2880-440f-8e99-4ab964ef0543","Type":"ContainerDied","Data":"c079c0faf93ecce647cd9aa5ad57a2a37936b5cde9f48da1ca73bdbcee862e3e"}
Sep 16 19:31:01.351689 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.351451    2565 scope.go:117] "RemoveContainer" containerID="7e0e453ab89d6d61fff9c61ed7085a5185392c039fd15672899dde84efe8810b"
Sep 16 19:31:01.351689 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.351397    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"
Sep 16 19:31:01.389530 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.389493    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"]
Sep 16 19:31:01.393214 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:01.393185    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8665d9d48c-5x4gq"]
Sep 16 19:31:02.751181 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:02.751143    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ca20bca-2880-440f-8e99-4ab964ef0543" path="/var/lib/kubelet/pods/5ca20bca-2880-440f-8e99-4ab964ef0543/volumes"
Sep 16 19:31:39.886297 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886210    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"]
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886492    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886504    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886516    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="storage-initializer"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886522    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="storage-initializer"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886537    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ca20bca-2880-440f-8e99-4ab964ef0543" containerName="storage-initializer"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886546    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ca20bca-2880-440f-8e99-4ab964ef0543" containerName="storage-initializer"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886554    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kube-rbac-proxy"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886560    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kube-rbac-proxy"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886569    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ca20bca-2880-440f-8e99-4ab964ef0543" containerName="storage-initializer"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886574    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ca20bca-2880-440f-8e99-4ab964ef0543" containerName="storage-initializer"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886621    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ca20bca-2880-440f-8e99-4ab964ef0543" containerName="storage-initializer"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886630    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kube-rbac-proxy"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886638    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ca20bca-2880-440f-8e99-4ab964ef0543" containerName="storage-initializer"
Sep 16 19:31:39.886682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.886643    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62b9056-3fcf-401f-bd88-99157c2f054b" containerName="kserve-container"
Sep 16 19:31:39.889662 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.889645    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:39.893543 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.893503    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 16 19:31:39.893543 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.893509    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:31:39.893543 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.893542    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:31:39.894134 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.894099    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:31:39.894219 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.894201    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 16 19:31:39.894281 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.894213    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:31:39.894281 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.894242    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:31:39.904320 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.904291    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"]
Sep 16 19:31:39.907276 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.907249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03c86426-2924-4140-b90d-b998fcab46e2-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:39.907422 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.907289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03c86426-2924-4140-b90d-b998fcab46e2-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:39.907422 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.907335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03c86426-2924-4140-b90d-b998fcab46e2-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:39.907422 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.907366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffqxq\" (UniqueName: \"kubernetes.io/projected/03c86426-2924-4140-b90d-b998fcab46e2-kube-api-access-ffqxq\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:39.907557 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:39.907441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03c86426-2924-4140-b90d-b998fcab46e2-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:40.008661 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:40.008628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03c86426-2924-4140-b90d-b998fcab46e2-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:40.008851 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:40.008669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03c86426-2924-4140-b90d-b998fcab46e2-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:40.008851 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:40.008709    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03c86426-2924-4140-b90d-b998fcab46e2-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:40.008851 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:40.008730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffqxq\" (UniqueName: \"kubernetes.io/projected/03c86426-2924-4140-b90d-b998fcab46e2-kube-api-access-ffqxq\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:40.008851 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:40.008765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03c86426-2924-4140-b90d-b998fcab46e2-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:40.009155 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:31:40.008872    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Sep 16 19:31:40.009155 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:31:40.008951    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c86426-2924-4140-b90d-b998fcab46e2-proxy-tls podName:03c86426-2924-4140-b90d-b998fcab46e2 nodeName:}" failed. No retries permitted until 2026-09-16 19:31:40.50893193 +0000 UTC m=+2784.322520784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/03c86426-2924-4140-b90d-b998fcab46e2-proxy-tls") pod "isvc-pmml-predictor-8674795f59-jt9wl" (UID: "03c86426-2924-4140-b90d-b998fcab46e2") : secret "isvc-pmml-predictor-serving-cert" not found
Sep 16 19:31:40.009261 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:40.009176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03c86426-2924-4140-b90d-b998fcab46e2-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:40.009369 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:40.009350    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03c86426-2924-4140-b90d-b998fcab46e2-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:40.009429 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:40.009412    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03c86426-2924-4140-b90d-b998fcab46e2-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:40.018237 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:40.018200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffqxq\" (UniqueName: \"kubernetes.io/projected/03c86426-2924-4140-b90d-b998fcab46e2-kube-api-access-ffqxq\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:40.513354 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:40.513317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03c86426-2924-4140-b90d-b998fcab46e2-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:40.515672 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:40.515642    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03c86426-2924-4140-b90d-b998fcab46e2-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-jt9wl\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:40.799704 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:40.799610    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:41.079416 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:41.079388    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"]
Sep 16 19:31:41.081787 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:31:41.081759    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c86426_2924_4140_b90d_b998fcab46e2.slice/crio-eef1a10cf919e57022b69a822dfd9b79c3481ab2fffec313c265f53cf6dd5ddc WatchSource:0}: Error finding container eef1a10cf919e57022b69a822dfd9b79c3481ab2fffec313c265f53cf6dd5ddc: Status 404 returned error can't find the container with id eef1a10cf919e57022b69a822dfd9b79c3481ab2fffec313c265f53cf6dd5ddc
Sep 16 19:31:41.496760 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:41.496720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" event={"ID":"03c86426-2924-4140-b90d-b998fcab46e2","Type":"ContainerStarted","Data":"b39edab5ea0ad473aa4baebae969021556c82c04e845e9cffe707f6b990c7047"}
Sep 16 19:31:41.496953 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:41.496794    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" event={"ID":"03c86426-2924-4140-b90d-b998fcab46e2","Type":"ContainerStarted","Data":"eef1a10cf919e57022b69a822dfd9b79c3481ab2fffec313c265f53cf6dd5ddc"}
Sep 16 19:31:45.513093 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:45.513056    2565 generic.go:356] "Generic (PLEG): container finished" podID="03c86426-2924-4140-b90d-b998fcab46e2" containerID="b39edab5ea0ad473aa4baebae969021556c82c04e845e9cffe707f6b990c7047" exitCode=0
Sep 16 19:31:45.513603 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:45.513135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" event={"ID":"03c86426-2924-4140-b90d-b998fcab46e2","Type":"ContainerDied","Data":"b39edab5ea0ad473aa4baebae969021556c82c04e845e9cffe707f6b990c7047"}
Sep 16 19:31:54.552380 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:54.552344    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" event={"ID":"03c86426-2924-4140-b90d-b998fcab46e2","Type":"ContainerStarted","Data":"b82a74f899d1dd6fbcb69537100159b05d860ef636fd64fd16f69c942221db6e"}
Sep 16 19:31:54.552772 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:54.552405    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" event={"ID":"03c86426-2924-4140-b90d-b998fcab46e2","Type":"ContainerStarted","Data":"fa5de812ece7fe958728a0d6472dc4e1a90493fe5815ee3def8fe00cd92db2f1"}
Sep 16 19:31:54.552772 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:54.552594    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:54.573841 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:54.573797    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podStartSLOduration=3.391419344 podStartE2EDuration="15.573783266s" totalImagesPullingTime="8.418197323s" totalInitContainerRuntime="3.764166599s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:54.57259691 +0000 UTC m=+2798.386185779" watchObservedRunningTime="2026-09-16 19:31:54.573783266 +0000 UTC m=+2798.387372125"
Sep 16 19:31:55.556458 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:55.556414    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:31:55.557982 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:55.557947    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:31:56.560198 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:31:56.560146    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:32:01.564899 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:32:01.564872    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:32:01.565580 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:32:01.565547    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:32:11.565711 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:32:11.565666    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:32:21.566088 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:32:21.566044    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:32:31.566159 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:32:31.566086    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:32:41.565710 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:32:41.565660    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:32:51.566300 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:32:51.566253    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:33:01.566503 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:01.566458    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:33:11.566349 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:11.566253    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:33:19.747469 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:19.747432    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:33:30.782461 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:30.782424    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"]
Sep 16 19:33:30.782902 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:30.782671    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container" containerID="cri-o://fa5de812ece7fe958728a0d6472dc4e1a90493fe5815ee3def8fe00cd92db2f1" gracePeriod=30
Sep 16 19:33:30.782902 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:30.782694    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kube-rbac-proxy" containerID="cri-o://b82a74f899d1dd6fbcb69537100159b05d860ef636fd64fd16f69c942221db6e" gracePeriod=30
Sep 16 19:33:30.902381 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:30.902348    2565 generic.go:356] "Generic (PLEG): container finished" podID="03c86426-2924-4140-b90d-b998fcab46e2" containerID="b82a74f899d1dd6fbcb69537100159b05d860ef636fd64fd16f69c942221db6e" exitCode=2
Sep 16 19:33:30.902560 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:30.902416    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" event={"ID":"03c86426-2924-4140-b90d-b998fcab46e2","Type":"ContainerDied","Data":"b82a74f899d1dd6fbcb69537100159b05d860ef636fd64fd16f69c942221db6e"}
Sep 16 19:33:31.561240 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:31.561194    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 19:33:34.919497 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:34.919460    2565 generic.go:356] "Generic (PLEG): container finished" podID="03c86426-2924-4140-b90d-b998fcab46e2" containerID="fa5de812ece7fe958728a0d6472dc4e1a90493fe5815ee3def8fe00cd92db2f1" exitCode=0
Sep 16 19:33:34.919867 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:34.919513    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" event={"ID":"03c86426-2924-4140-b90d-b998fcab46e2","Type":"ContainerDied","Data":"fa5de812ece7fe958728a0d6472dc4e1a90493fe5815ee3def8fe00cd92db2f1"}
Sep 16 19:33:35.017922 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.017895    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:33:35.134451 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.134358    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03c86426-2924-4140-b90d-b998fcab46e2-proxy-tls\") pod \"03c86426-2924-4140-b90d-b998fcab46e2\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") "
Sep 16 19:33:35.134451 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.134422    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03c86426-2924-4140-b90d-b998fcab46e2-kube-rbac-proxy-sar-config\") pod \"03c86426-2924-4140-b90d-b998fcab46e2\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") "
Sep 16 19:33:35.134451 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.134440    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffqxq\" (UniqueName: \"kubernetes.io/projected/03c86426-2924-4140-b90d-b998fcab46e2-kube-api-access-ffqxq\") pod \"03c86426-2924-4140-b90d-b998fcab46e2\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") "
Sep 16 19:33:35.134714 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.134474    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03c86426-2924-4140-b90d-b998fcab46e2-cabundle-cert\") pod \"03c86426-2924-4140-b90d-b998fcab46e2\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") "
Sep 16 19:33:35.134714 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.134521    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03c86426-2924-4140-b90d-b998fcab46e2-kserve-provision-location\") pod \"03c86426-2924-4140-b90d-b998fcab46e2\" (UID: \"03c86426-2924-4140-b90d-b998fcab46e2\") "
Sep 16 19:33:35.134871 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.134837    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c86426-2924-4140-b90d-b998fcab46e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03c86426-2924-4140-b90d-b998fcab46e2" (UID: "03c86426-2924-4140-b90d-b998fcab46e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:35.134938 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.134864    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c86426-2924-4140-b90d-b998fcab46e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "03c86426-2924-4140-b90d-b998fcab46e2" (UID: "03c86426-2924-4140-b90d-b998fcab46e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:35.134938 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.134913    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c86426-2924-4140-b90d-b998fcab46e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "03c86426-2924-4140-b90d-b998fcab46e2" (UID: "03c86426-2924-4140-b90d-b998fcab46e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:35.136696 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.136673    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c86426-2924-4140-b90d-b998fcab46e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03c86426-2924-4140-b90d-b998fcab46e2" (UID: "03c86426-2924-4140-b90d-b998fcab46e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:35.136696 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.136679    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c86426-2924-4140-b90d-b998fcab46e2-kube-api-access-ffqxq" (OuterVolumeSpecName: "kube-api-access-ffqxq") pod "03c86426-2924-4140-b90d-b998fcab46e2" (UID: "03c86426-2924-4140-b90d-b998fcab46e2"). InnerVolumeSpecName "kube-api-access-ffqxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:35.235304 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.235260    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03c86426-2924-4140-b90d-b998fcab46e2-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:33:35.235304 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.235297    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03c86426-2924-4140-b90d-b998fcab46e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:33:35.235304 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.235310    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffqxq\" (UniqueName: \"kubernetes.io/projected/03c86426-2924-4140-b90d-b998fcab46e2-kube-api-access-ffqxq\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:33:35.235542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.235319    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03c86426-2924-4140-b90d-b998fcab46e2-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:33:35.235542 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.235329    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03c86426-2924-4140-b90d-b998fcab46e2-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:33:35.925199 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.925165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl" event={"ID":"03c86426-2924-4140-b90d-b998fcab46e2","Type":"ContainerDied","Data":"eef1a10cf919e57022b69a822dfd9b79c3481ab2fffec313c265f53cf6dd5ddc"}
Sep 16 19:33:35.925650 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.925234    2565 scope.go:117] "RemoveContainer" containerID="b82a74f899d1dd6fbcb69537100159b05d860ef636fd64fd16f69c942221db6e"
Sep 16 19:33:35.925650 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.925239    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"
Sep 16 19:33:35.934496 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.934458    2565 scope.go:117] "RemoveContainer" containerID="fa5de812ece7fe958728a0d6472dc4e1a90493fe5815ee3def8fe00cd92db2f1"
Sep 16 19:33:35.942504 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.942485    2565 scope.go:117] "RemoveContainer" containerID="b39edab5ea0ad473aa4baebae969021556c82c04e845e9cffe707f6b990c7047"
Sep 16 19:33:35.950590 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.950564    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"]
Sep 16 19:33:35.952178 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.952151    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jt9wl"]
Sep 16 19:33:35.974852 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.974820    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"]
Sep 16 19:33:35.975199 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.975182    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="storage-initializer"
Sep 16 19:33:35.975290 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.975202    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="storage-initializer"
Sep 16 19:33:35.975290 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.975227    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container"
Sep 16 19:33:35.975290 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.975237    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container"
Sep 16 19:33:35.975290 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.975247    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kube-rbac-proxy"
Sep 16 19:33:35.975290 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.975255    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kube-rbac-proxy"
Sep 16 19:33:35.975539 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.975330    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kube-rbac-proxy"
Sep 16 19:33:35.975539 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.975345    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="03c86426-2924-4140-b90d-b998fcab46e2" containerName="kserve-container"
Sep 16 19:33:35.979965 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.979939    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:35.982696 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.982668    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 16 19:33:35.982824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.982748    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:33:35.982824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.982757    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:33:35.982824 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.982807    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:33:35.983490 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.983467    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:33:35.983599 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.983526    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:33:35.983654 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.983628    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:33:35.992146 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:35.992092    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"]
Sep 16 19:33:36.142339 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.142292    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.142339 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.142344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbkkb\" (UniqueName: \"kubernetes.io/projected/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kube-api-access-cbkkb\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.142631 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.142383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c0ec3c7-c248-45e0-a4d9-331380cb8518-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.142631 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.142463    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0ec3c7-c248-45e0-a4d9-331380cb8518-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.142631 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.142508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.243487 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.243437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c0ec3c7-c248-45e0-a4d9-331380cb8518-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.243487 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.243500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0ec3c7-c248-45e0-a4d9-331380cb8518-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.243758 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.243526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.243758 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.243552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.243758 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.243574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbkkb\" (UniqueName: \"kubernetes.io/projected/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kube-api-access-cbkkb\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.243758 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:33:36.243697    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 16 19:33:36.243944 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:33:36.243771    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c0ec3c7-c248-45e0-a4d9-331380cb8518-proxy-tls podName:0c0ec3c7-c248-45e0-a4d9-331380cb8518 nodeName:}" failed. No retries permitted until 2026-09-16 19:33:36.743751543 +0000 UTC m=+2900.557340380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0c0ec3c7-c248-45e0-a4d9-331380cb8518-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-t5bnx" (UID: "0c0ec3c7-c248-45e0-a4d9-331380cb8518") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 16 19:33:36.243989 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.243960    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.244253 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.244235    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c0ec3c7-c248-45e0-a4d9-331380cb8518-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.244313 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.244262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.252952 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.252930    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbkkb\" (UniqueName: \"kubernetes.io/projected/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kube-api-access-cbkkb\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.747817 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.747778    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0ec3c7-c248-45e0-a4d9-331380cb8518-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.750677 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.750649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0ec3c7-c248-45e0-a4d9-331380cb8518-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-t5bnx\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:36.751258 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.751236    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03c86426-2924-4140-b90d-b998fcab46e2" path="/var/lib/kubelet/pods/03c86426-2924-4140-b90d-b998fcab46e2/volumes"
Sep 16 19:33:36.890415 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:36.890375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:37.163716 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:37.163633    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"]
Sep 16 19:33:37.166803 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:33:37.166773    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0ec3c7_c248_45e0_a4d9_331380cb8518.slice/crio-5301217e283944fcc8d5e4f1edcebd0de8bdd855b43a030c3ec4427347b38b18 WatchSource:0}: Error finding container 5301217e283944fcc8d5e4f1edcebd0de8bdd855b43a030c3ec4427347b38b18: Status 404 returned error can't find the container with id 5301217e283944fcc8d5e4f1edcebd0de8bdd855b43a030c3ec4427347b38b18
Sep 16 19:33:37.935589 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:37.935551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" event={"ID":"0c0ec3c7-c248-45e0-a4d9-331380cb8518","Type":"ContainerStarted","Data":"6d1eeb300aa919b6b53a04d5c40e78aefe1792322cb491b157de776cc53a14cd"}
Sep 16 19:33:37.935790 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:37.935630    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" event={"ID":"0c0ec3c7-c248-45e0-a4d9-331380cb8518","Type":"ContainerStarted","Data":"5301217e283944fcc8d5e4f1edcebd0de8bdd855b43a030c3ec4427347b38b18"}
Sep 16 19:33:41.950846 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:41.950810    2565 generic.go:356] "Generic (PLEG): container finished" podID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerID="6d1eeb300aa919b6b53a04d5c40e78aefe1792322cb491b157de776cc53a14cd" exitCode=0
Sep 16 19:33:41.951355 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:41.950868    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" event={"ID":"0c0ec3c7-c248-45e0-a4d9-331380cb8518","Type":"ContainerDied","Data":"6d1eeb300aa919b6b53a04d5c40e78aefe1792322cb491b157de776cc53a14cd"}
Sep 16 19:33:42.956939 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:42.956903    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" event={"ID":"0c0ec3c7-c248-45e0-a4d9-331380cb8518","Type":"ContainerStarted","Data":"95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04"}
Sep 16 19:33:42.957371 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:42.956968    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" event={"ID":"0c0ec3c7-c248-45e0-a4d9-331380cb8518","Type":"ContainerStarted","Data":"c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f"}
Sep 16 19:33:42.957371 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:42.957163    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:42.977365 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:42.977308    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podStartSLOduration=4.007290339 podStartE2EDuration="7.977290927s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.970000588s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:42.976031732 +0000 UTC m=+2906.789620592" watchObservedRunningTime="2026-09-16 19:33:42.977290927 +0000 UTC m=+2906.790879787"
Sep 16 19:33:43.960832 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:43.960799    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:43.962137 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:43.962096    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:33:44.964699 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:44.964659    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:33:49.968682 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:49.968654    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:33:49.969089 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:49.969042    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:33:59.971204 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:33:59.971151    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:34:06.537702 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.537667    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"]
Sep 16 19:34:06.540931 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.540914    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.543564 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.543537    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-1c9c8b-kube-rbac-proxy-sar-config\""
Sep 16 19:34:06.543689 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.543575    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-1c9c8b-predictor-serving-cert\""
Sep 16 19:34:06.550200 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.550171    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"]
Sep 16 19:34:06.579667 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.579640    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81621736-37f1-4a1c-be17-6878793b7078-kube-rbac-proxy-sar-config\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.579806 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.579675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81621736-37f1-4a1c-be17-6878793b7078-cabundle-cert\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.579806 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.579748    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81621736-37f1-4a1c-be17-6878793b7078-kserve-provision-location\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.579806 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.579790    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81621736-37f1-4a1c-be17-6878793b7078-proxy-tls\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.579936 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.579831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8zsr\" (UniqueName: \"kubernetes.io/projected/81621736-37f1-4a1c-be17-6878793b7078-kube-api-access-d8zsr\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.680772 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.680735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81621736-37f1-4a1c-be17-6878793b7078-kube-rbac-proxy-sar-config\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.680772 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.680772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81621736-37f1-4a1c-be17-6878793b7078-cabundle-cert\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.681014 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.680825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81621736-37f1-4a1c-be17-6878793b7078-kserve-provision-location\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.681014 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.680855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81621736-37f1-4a1c-be17-6878793b7078-proxy-tls\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.681014 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.680997    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8zsr\" (UniqueName: \"kubernetes.io/projected/81621736-37f1-4a1c-be17-6878793b7078-kube-api-access-d8zsr\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.681387 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.681362    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81621736-37f1-4a1c-be17-6878793b7078-kserve-provision-location\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.681560 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.681541    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81621736-37f1-4a1c-be17-6878793b7078-cabundle-cert\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.681608 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.681585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81621736-37f1-4a1c-be17-6878793b7078-kube-rbac-proxy-sar-config\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.683371 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.683354    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81621736-37f1-4a1c-be17-6878793b7078-proxy-tls\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.689152 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.689130    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8zsr\" (UniqueName: \"kubernetes.io/projected/81621736-37f1-4a1c-be17-6878793b7078-kube-api-access-d8zsr\") pod \"isvc-primary-1c9c8b-predictor-745dbb8587-46drw\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") " pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:06.850955 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:06.850853    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:07.126269 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:07.126239    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"]
Sep 16 19:34:07.128883 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:34:07.128845    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81621736_37f1_4a1c_be17_6878793b7078.slice/crio-f069910f7abad32fa80c204443ff35dea1da002bdf6a4f1bf9802c51c5b09556 WatchSource:0}: Error finding container f069910f7abad32fa80c204443ff35dea1da002bdf6a4f1bf9802c51c5b09556: Status 404 returned error can't find the container with id f069910f7abad32fa80c204443ff35dea1da002bdf6a4f1bf9802c51c5b09556
Sep 16 19:34:08.050440 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:08.050401    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" event={"ID":"81621736-37f1-4a1c-be17-6878793b7078","Type":"ContainerStarted","Data":"3e49f5931dd297e425e2414b00cfbfbe72ceac39fbf06ae9843e7eed5c57c52d"}
Sep 16 19:34:08.050831 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:08.050492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" event={"ID":"81621736-37f1-4a1c-be17-6878793b7078","Type":"ContainerStarted","Data":"f069910f7abad32fa80c204443ff35dea1da002bdf6a4f1bf9802c51c5b09556"}
Sep 16 19:34:09.969592 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:09.969551    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:34:12.067765 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:12.067723    2565 generic.go:356] "Generic (PLEG): container finished" podID="81621736-37f1-4a1c-be17-6878793b7078" containerID="3e49f5931dd297e425e2414b00cfbfbe72ceac39fbf06ae9843e7eed5c57c52d" exitCode=0
Sep 16 19:34:12.068266 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:12.067803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" event={"ID":"81621736-37f1-4a1c-be17-6878793b7078","Type":"ContainerDied","Data":"3e49f5931dd297e425e2414b00cfbfbe72ceac39fbf06ae9843e7eed5c57c52d"}
Sep 16 19:34:13.074689 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:13.074652    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" event={"ID":"81621736-37f1-4a1c-be17-6878793b7078","Type":"ContainerStarted","Data":"805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f"}
Sep 16 19:34:13.075152 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:13.074720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" event={"ID":"81621736-37f1-4a1c-be17-6878793b7078","Type":"ContainerStarted","Data":"3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950"}
Sep 16 19:34:13.075152 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:13.075019    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:13.075152 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:13.075085    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:13.076712 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:13.076680    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:34:13.095858 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:13.095799    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podStartSLOduration=3.123075038 podStartE2EDuration="7.095774466s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.972699428s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:13.09496378 +0000 UTC m=+2936.908552640" watchObservedRunningTime="2026-09-16 19:34:13.095774466 +0000 UTC m=+2936.909363319"
Sep 16 19:34:14.078732 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:14.078692    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:34:19.083977 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:19.083940    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:34:19.084561 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:19.084531    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:34:19.969886 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:19.969841    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:34:29.084654 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:29.084613    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:34:29.969215 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:29.969173    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:34:39.084879 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:39.084839    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:34:39.970000 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:39.969896    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:34:49.084918 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:49.084846    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:34:49.969223 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:49.969181    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:34:54.747147 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:54.747074    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:34:59.085115 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:34:59.085070    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:35:04.747319 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:04.747274    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:35:09.084570 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:09.084528    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:35:14.752697 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:14.752664    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:35:17.130073 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:17.130030    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"]
Sep 16 19:35:17.130510 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:17.130341    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container" containerID="cri-o://c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f" gracePeriod=30
Sep 16 19:35:17.130510 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:17.130421    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kube-rbac-proxy" containerID="cri-o://95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04" gracePeriod=30
Sep 16 19:35:17.331178 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:17.331142    2565 generic.go:356] "Generic (PLEG): container finished" podID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerID="95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04" exitCode=2
Sep 16 19:35:17.331371 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:17.331207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" event={"ID":"0c0ec3c7-c248-45e0-a4d9-331380cb8518","Type":"ContainerDied","Data":"95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04"}
Sep 16 19:35:19.085028 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:19.084991    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:35:19.965957 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:19.965907    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 16 19:35:21.282691 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.282663    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:35:21.349124 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.348992    2565 generic.go:356] "Generic (PLEG): container finished" podID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerID="c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f" exitCode=0
Sep 16 19:35:21.349124 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.349043    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" event={"ID":"0c0ec3c7-c248-45e0-a4d9-331380cb8518","Type":"ContainerDied","Data":"c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f"}
Sep 16 19:35:21.349364 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.349124    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"
Sep 16 19:35:21.349364 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.349140    2565 scope.go:117] "RemoveContainer" containerID="95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04"
Sep 16 19:35:21.349364 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.349126    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx" event={"ID":"0c0ec3c7-c248-45e0-a4d9-331380cb8518","Type":"ContainerDied","Data":"5301217e283944fcc8d5e4f1edcebd0de8bdd855b43a030c3ec4427347b38b18"}
Sep 16 19:35:21.357239 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.357218    2565 scope.go:117] "RemoveContainer" containerID="c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f"
Sep 16 19:35:21.363428 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.363398    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbkkb\" (UniqueName: \"kubernetes.io/projected/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kube-api-access-cbkkb\") pod \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") "
Sep 16 19:35:21.363539 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.363447    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kserve-provision-location\") pod \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") "
Sep 16 19:35:21.363539 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.363485    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0ec3c7-c248-45e0-a4d9-331380cb8518-proxy-tls\") pod \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") "
Sep 16 19:35:21.363636 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.363580    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c0ec3c7-c248-45e0-a4d9-331380cb8518-cabundle-cert\") pod \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") "
Sep 16 19:35:21.363636 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.363626    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kube-rbac-proxy-sar-config\") pod \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\" (UID: \"0c0ec3c7-c248-45e0-a4d9-331380cb8518\") "
Sep 16 19:35:21.363787 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.363761    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c0ec3c7-c248-45e0-a4d9-331380cb8518" (UID: "0c0ec3c7-c248-45e0-a4d9-331380cb8518"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:21.363951 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.363929    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:21.364062 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.364008    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0ec3c7-c248-45e0-a4d9-331380cb8518-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0c0ec3c7-c248-45e0-a4d9-331380cb8518" (UID: "0c0ec3c7-c248-45e0-a4d9-331380cb8518"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:21.364137 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.364060    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0c0ec3c7-c248-45e0-a4d9-331380cb8518" (UID: "0c0ec3c7-c248-45e0-a4d9-331380cb8518"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:21.365091 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.365063    2565 scope.go:117] "RemoveContainer" containerID="6d1eeb300aa919b6b53a04d5c40e78aefe1792322cb491b157de776cc53a14cd"
Sep 16 19:35:21.365710 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.365682    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kube-api-access-cbkkb" (OuterVolumeSpecName: "kube-api-access-cbkkb") pod "0c0ec3c7-c248-45e0-a4d9-331380cb8518" (UID: "0c0ec3c7-c248-45e0-a4d9-331380cb8518"). InnerVolumeSpecName "kube-api-access-cbkkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:21.365797 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.365746    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0ec3c7-c248-45e0-a4d9-331380cb8518-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c0ec3c7-c248-45e0-a4d9-331380cb8518" (UID: "0c0ec3c7-c248-45e0-a4d9-331380cb8518"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:21.378808 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.378788    2565 scope.go:117] "RemoveContainer" containerID="95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04"
Sep 16 19:35:21.379248 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:35:21.379228    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04\": container with ID starting with 95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04 not found: ID does not exist" containerID="95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04"
Sep 16 19:35:21.379308 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.379264    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04"} err="failed to get container status \"95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04\": rpc error: code = NotFound desc = could not find container \"95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04\": container with ID starting with 95c57a7c5d24e62cae569a0d8290ad80c880d293f12359b9aebb5dbfc81a5e04 not found: ID does not exist"
Sep 16 19:35:21.379308 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.379283    2565 scope.go:117] "RemoveContainer" containerID="c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f"
Sep 16 19:35:21.379549 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:35:21.379535    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f\": container with ID starting with c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f not found: ID does not exist" containerID="c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f"
Sep 16 19:35:21.379595 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.379552    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f"} err="failed to get container status \"c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f\": rpc error: code = NotFound desc = could not find container \"c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f\": container with ID starting with c7fc248be715eb7187d77ed0d8111707daaa6ba2a8b29959432a85316cb2dc5f not found: ID does not exist"
Sep 16 19:35:21.379595 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.379569    2565 scope.go:117] "RemoveContainer" containerID="6d1eeb300aa919b6b53a04d5c40e78aefe1792322cb491b157de776cc53a14cd"
Sep 16 19:35:21.379834 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:35:21.379812    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1eeb300aa919b6b53a04d5c40e78aefe1792322cb491b157de776cc53a14cd\": container with ID starting with 6d1eeb300aa919b6b53a04d5c40e78aefe1792322cb491b157de776cc53a14cd not found: ID does not exist" containerID="6d1eeb300aa919b6b53a04d5c40e78aefe1792322cb491b157de776cc53a14cd"
Sep 16 19:35:21.380010 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.379841    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1eeb300aa919b6b53a04d5c40e78aefe1792322cb491b157de776cc53a14cd"} err="failed to get container status \"6d1eeb300aa919b6b53a04d5c40e78aefe1792322cb491b157de776cc53a14cd\": rpc error: code = NotFound desc = could not find container \"6d1eeb300aa919b6b53a04d5c40e78aefe1792322cb491b157de776cc53a14cd\": container with ID starting with 6d1eeb300aa919b6b53a04d5c40e78aefe1792322cb491b157de776cc53a14cd not found: ID does not exist"
Sep 16 19:35:21.464947 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.464911    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c0ec3c7-c248-45e0-a4d9-331380cb8518-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:21.464947 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.464940    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:21.464947 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.464950    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbkkb\" (UniqueName: \"kubernetes.io/projected/0c0ec3c7-c248-45e0-a4d9-331380cb8518-kube-api-access-cbkkb\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:21.465193 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.464959    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0ec3c7-c248-45e0-a4d9-331380cb8518-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:35:21.672531 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.672499    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"]
Sep 16 19:35:21.676878 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:21.676851    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-t5bnx"]
Sep 16 19:35:22.751837 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:22.751791    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" path="/var/lib/kubelet/pods/0c0ec3c7-c248-45e0-a4d9-331380cb8518/volumes"
Sep 16 19:35:29.085121 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:29.085066    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:35:39.085627 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:35:39.085583    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:36:01.716320 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.716281    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"]
Sep 16 19:36:01.716847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.716606    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="storage-initializer"
Sep 16 19:36:01.716847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.716624    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="storage-initializer"
Sep 16 19:36:01.716847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.716645    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container"
Sep 16 19:36:01.716847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.716654    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container"
Sep 16 19:36:01.716847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.716665    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kube-rbac-proxy"
Sep 16 19:36:01.716847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.716674    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kube-rbac-proxy"
Sep 16 19:36:01.716847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.716745    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kserve-container"
Sep 16 19:36:01.716847 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.716756    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c0ec3c7-c248-45e0-a4d9-331380cb8518" containerName="kube-rbac-proxy"
Sep 16 19:36:01.719775 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.719758    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.722581 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.722550    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-1c9c8b-predictor-serving-cert\""
Sep 16 19:36:01.722718 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.722558    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-1c9c8b\""
Sep 16 19:36:01.722894 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.722877    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-1c9c8b-kube-rbac-proxy-sar-config\""
Sep 16 19:36:01.722953 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.722909    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-1c9c8b-dockercfg-qlk2t\""
Sep 16 19:36:01.731168 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.731135    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"]
Sep 16 19:36:01.892360 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.892318    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.892360 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.892361    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gj6v\" (UniqueName: \"kubernetes.io/projected/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kube-api-access-2gj6v\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.892603 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.892397    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kserve-provision-location\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.892603 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.892572    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b741a3d6-5728-4861-bd3e-c90ef033b4b7-cabundle-cert\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.892760 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.892732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b741a3d6-5728-4861-bd3e-c90ef033b4b7-proxy-tls\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.993522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.993418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b741a3d6-5728-4861-bd3e-c90ef033b4b7-proxy-tls\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.993522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.993468    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.993522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.993488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gj6v\" (UniqueName: \"kubernetes.io/projected/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kube-api-access-2gj6v\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.993522 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.993508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kserve-provision-location\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.993911 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.993550    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b741a3d6-5728-4861-bd3e-c90ef033b4b7-cabundle-cert\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.994047 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.994020    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kserve-provision-location\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.994234 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.994214    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b741a3d6-5728-4861-bd3e-c90ef033b4b7-cabundle-cert\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.994327 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.994308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:01.996030 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:01.996006    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b741a3d6-5728-4861-bd3e-c90ef033b4b7-proxy-tls\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:02.002668 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:02.002628    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gj6v\" (UniqueName: \"kubernetes.io/projected/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kube-api-access-2gj6v\") pod \"isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") " pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:02.030328 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:02.030290    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:02.304795 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:02.304623    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"]
Sep 16 19:36:02.307134 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:36:02.307090    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb741a3d6_5728_4861_bd3e_c90ef033b4b7.slice/crio-60b044569c9bdbc43f9d5b9587a2543572671e031b2fbcc20dec9f4b314b9ae4 WatchSource:0}: Error finding container 60b044569c9bdbc43f9d5b9587a2543572671e031b2fbcc20dec9f4b314b9ae4: Status 404 returned error can't find the container with id 60b044569c9bdbc43f9d5b9587a2543572671e031b2fbcc20dec9f4b314b9ae4
Sep 16 19:36:02.507515 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:02.507479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp" event={"ID":"b741a3d6-5728-4861-bd3e-c90ef033b4b7","Type":"ContainerStarted","Data":"f1e9e7bdbcf02dfb9944d8b84588809f32d6c82ec5b9fb77775a72f894cc1700"}
Sep 16 19:36:02.507698 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:02.507550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp" event={"ID":"b741a3d6-5728-4861-bd3e-c90ef033b4b7","Type":"ContainerStarted","Data":"60b044569c9bdbc43f9d5b9587a2543572671e031b2fbcc20dec9f4b314b9ae4"}
Sep 16 19:36:07.527958 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:07.527925    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp_b741a3d6-5728-4861-bd3e-c90ef033b4b7/storage-initializer/0.log"
Sep 16 19:36:07.528382 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:07.527977    2565 generic.go:356] "Generic (PLEG): container finished" podID="b741a3d6-5728-4861-bd3e-c90ef033b4b7" containerID="f1e9e7bdbcf02dfb9944d8b84588809f32d6c82ec5b9fb77775a72f894cc1700" exitCode=1
Sep 16 19:36:07.528382 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:07.528057    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp" event={"ID":"b741a3d6-5728-4861-bd3e-c90ef033b4b7","Type":"ContainerDied","Data":"f1e9e7bdbcf02dfb9944d8b84588809f32d6c82ec5b9fb77775a72f894cc1700"}
Sep 16 19:36:08.533649 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:08.533622    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp_b741a3d6-5728-4861-bd3e-c90ef033b4b7/storage-initializer/0.log"
Sep 16 19:36:08.534006 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:08.533719    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp" event={"ID":"b741a3d6-5728-4861-bd3e-c90ef033b4b7","Type":"ContainerStarted","Data":"c22a3356aa9777dfcef143e53e4c4bf4b62fff79e0c4939c4bf8dc07a267f0a8"}
Sep 16 19:36:10.545198 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:10.545167    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp_b741a3d6-5728-4861-bd3e-c90ef033b4b7/storage-initializer/1.log"
Sep 16 19:36:10.545654 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:10.545640    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp_b741a3d6-5728-4861-bd3e-c90ef033b4b7/storage-initializer/0.log"
Sep 16 19:36:10.545727 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:10.545684    2565 generic.go:356] "Generic (PLEG): container finished" podID="b741a3d6-5728-4861-bd3e-c90ef033b4b7" containerID="c22a3356aa9777dfcef143e53e4c4bf4b62fff79e0c4939c4bf8dc07a267f0a8" exitCode=1
Sep 16 19:36:10.545793 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:10.545767    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp" event={"ID":"b741a3d6-5728-4861-bd3e-c90ef033b4b7","Type":"ContainerDied","Data":"c22a3356aa9777dfcef143e53e4c4bf4b62fff79e0c4939c4bf8dc07a267f0a8"}
Sep 16 19:36:10.545858 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:10.545845    2565 scope.go:117] "RemoveContainer" containerID="f1e9e7bdbcf02dfb9944d8b84588809f32d6c82ec5b9fb77775a72f894cc1700"
Sep 16 19:36:10.546254 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:10.546237    2565 scope.go:117] "RemoveContainer" containerID="f1e9e7bdbcf02dfb9944d8b84588809f32d6c82ec5b9fb77775a72f894cc1700"
Sep 16 19:36:10.558005 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:36:10.557962    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp_kserve-ci-e2e-test(b741a3d6-5728-4861-bd3e-c90ef033b4b7)\"" pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp" podUID="b741a3d6-5728-4861-bd3e-c90ef033b4b7"
Sep 16 19:36:10.561000 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:36:10.560959    2565 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp_kserve-ci-e2e-test_b741a3d6-5728-4861-bd3e-c90ef033b4b7_0 in pod sandbox 60b044569c9bdbc43f9d5b9587a2543572671e031b2fbcc20dec9f4b314b9ae4 from index: no such id: 'f1e9e7bdbcf02dfb9944d8b84588809f32d6c82ec5b9fb77775a72f894cc1700'" containerID="f1e9e7bdbcf02dfb9944d8b84588809f32d6c82ec5b9fb77775a72f894cc1700"
Sep 16 19:36:10.561270 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:10.561238    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e9e7bdbcf02dfb9944d8b84588809f32d6c82ec5b9fb77775a72f894cc1700"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp_kserve-ci-e2e-test_b741a3d6-5728-4861-bd3e-c90ef033b4b7_0 in pod sandbox 60b044569c9bdbc43f9d5b9587a2543572671e031b2fbcc20dec9f4b314b9ae4 from index: no such id: 'f1e9e7bdbcf02dfb9944d8b84588809f32d6c82ec5b9fb77775a72f894cc1700'"
Sep 16 19:36:11.557713 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:11.557677    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp_b741a3d6-5728-4861-bd3e-c90ef033b4b7/storage-initializer/1.log"
Sep 16 19:36:19.751387 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.751344    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"]
Sep 16 19:36:19.792578 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.792530    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"]
Sep 16 19:36:19.792941 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.792884    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container" containerID="cri-o://3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950" gracePeriod=30
Sep 16 19:36:19.793022 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.792914    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kube-rbac-proxy" containerID="cri-o://805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f" gracePeriod=30
Sep 16 19:36:19.911255 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.911233    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp_b741a3d6-5728-4861-bd3e-c90ef033b4b7/storage-initializer/1.log"
Sep 16 19:36:19.913295 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.913278    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:19.932932 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.932902    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kserve-provision-location\") pod \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") "
Sep 16 19:36:19.932932 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.932948    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kube-rbac-proxy-sar-config\") pod \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") "
Sep 16 19:36:19.933153 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.932996    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gj6v\" (UniqueName: \"kubernetes.io/projected/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kube-api-access-2gj6v\") pod \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") "
Sep 16 19:36:19.933153 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.933037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b741a3d6-5728-4861-bd3e-c90ef033b4b7-proxy-tls\") pod \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") "
Sep 16 19:36:19.933153 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.933080    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b741a3d6-5728-4861-bd3e-c90ef033b4b7-cabundle-cert\") pod \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\" (UID: \"b741a3d6-5728-4861-bd3e-c90ef033b4b7\") "
Sep 16 19:36:19.933311 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.933200    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b741a3d6-5728-4861-bd3e-c90ef033b4b7" (UID: "b741a3d6-5728-4861-bd3e-c90ef033b4b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:19.933372 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.933335    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:36:19.933442 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.933391    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b741a3d6-5728-4861-bd3e-c90ef033b4b7" (UID: "b741a3d6-5728-4861-bd3e-c90ef033b4b7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:36:19.933540 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.933515    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b741a3d6-5728-4861-bd3e-c90ef033b4b7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b741a3d6-5728-4861-bd3e-c90ef033b4b7" (UID: "b741a3d6-5728-4861-bd3e-c90ef033b4b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:36:19.935607 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.935532    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kube-api-access-2gj6v" (OuterVolumeSpecName: "kube-api-access-2gj6v") pod "b741a3d6-5728-4861-bd3e-c90ef033b4b7" (UID: "b741a3d6-5728-4861-bd3e-c90ef033b4b7"). InnerVolumeSpecName "kube-api-access-2gj6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:36:19.935718 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:19.935627    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b741a3d6-5728-4861-bd3e-c90ef033b4b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b741a3d6-5728-4861-bd3e-c90ef033b4b7" (UID: "b741a3d6-5728-4861-bd3e-c90ef033b4b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:36:20.033949 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.033842    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b741a3d6-5728-4861-bd3e-c90ef033b4b7-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:36:20.033949 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.033881    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b741a3d6-5728-4861-bd3e-c90ef033b4b7-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:36:20.033949 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.033896    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:36:20.033949 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.033909    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gj6v\" (UniqueName: \"kubernetes.io/projected/b741a3d6-5728-4861-bd3e-c90ef033b4b7-kube-api-access-2gj6v\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:36:20.596559 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.596525    2565 generic.go:356] "Generic (PLEG): container finished" podID="81621736-37f1-4a1c-be17-6878793b7078" containerID="805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f" exitCode=2
Sep 16 19:36:20.596762 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.596595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" event={"ID":"81621736-37f1-4a1c-be17-6878793b7078","Type":"ContainerDied","Data":"805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f"}
Sep 16 19:36:20.597929 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.597904    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp_b741a3d6-5728-4861-bd3e-c90ef033b4b7/storage-initializer/1.log"
Sep 16 19:36:20.598094 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.598007    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"
Sep 16 19:36:20.598094 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.598010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp" event={"ID":"b741a3d6-5728-4861-bd3e-c90ef033b4b7","Type":"ContainerDied","Data":"60b044569c9bdbc43f9d5b9587a2543572671e031b2fbcc20dec9f4b314b9ae4"}
Sep 16 19:36:20.598094 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.598082    2565 scope.go:117] "RemoveContainer" containerID="c22a3356aa9777dfcef143e53e4c4bf4b62fff79e0c4939c4bf8dc07a267f0a8"
Sep 16 19:36:20.640623 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.640590    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"]
Sep 16 19:36:20.644851 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.644826    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1c9c8b-predictor-cf8c7f9bd-lntrp"]
Sep 16 19:36:20.751062 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:20.751019    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b741a3d6-5728-4861-bd3e-c90ef033b4b7" path="/var/lib/kubelet/pods/b741a3d6-5728-4861-bd3e-c90ef033b4b7/volumes"
Sep 16 19:36:24.079674 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:24.079607    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 16 19:36:25.143910 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.143875    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:36:25.174287 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.174228    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81621736-37f1-4a1c-be17-6878793b7078-proxy-tls\") pod \"81621736-37f1-4a1c-be17-6878793b7078\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") "
Sep 16 19:36:25.174287 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.174290    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81621736-37f1-4a1c-be17-6878793b7078-cabundle-cert\") pod \"81621736-37f1-4a1c-be17-6878793b7078\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") "
Sep 16 19:36:25.174686 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.174363    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81621736-37f1-4a1c-be17-6878793b7078-kserve-provision-location\") pod \"81621736-37f1-4a1c-be17-6878793b7078\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") "
Sep 16 19:36:25.174686 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.174422    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8zsr\" (UniqueName: \"kubernetes.io/projected/81621736-37f1-4a1c-be17-6878793b7078-kube-api-access-d8zsr\") pod \"81621736-37f1-4a1c-be17-6878793b7078\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") "
Sep 16 19:36:25.174686 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.174470    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81621736-37f1-4a1c-be17-6878793b7078-kube-rbac-proxy-sar-config\") pod \"81621736-37f1-4a1c-be17-6878793b7078\" (UID: \"81621736-37f1-4a1c-be17-6878793b7078\") "
Sep 16 19:36:25.174880 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.174834    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81621736-37f1-4a1c-be17-6878793b7078-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81621736-37f1-4a1c-be17-6878793b7078" (UID: "81621736-37f1-4a1c-be17-6878793b7078"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:25.174963 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.174930    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81621736-37f1-4a1c-be17-6878793b7078-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81621736-37f1-4a1c-be17-6878793b7078" (UID: "81621736-37f1-4a1c-be17-6878793b7078"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:36:25.175032 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.174939    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81621736-37f1-4a1c-be17-6878793b7078-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81621736-37f1-4a1c-be17-6878793b7078" (UID: "81621736-37f1-4a1c-be17-6878793b7078"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:36:25.177156 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.177083    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81621736-37f1-4a1c-be17-6878793b7078-kube-api-access-d8zsr" (OuterVolumeSpecName: "kube-api-access-d8zsr") pod "81621736-37f1-4a1c-be17-6878793b7078" (UID: "81621736-37f1-4a1c-be17-6878793b7078"). InnerVolumeSpecName "kube-api-access-d8zsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:36:25.177156 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.177076    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81621736-37f1-4a1c-be17-6878793b7078-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81621736-37f1-4a1c-be17-6878793b7078" (UID: "81621736-37f1-4a1c-be17-6878793b7078"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:36:25.275231 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.275137    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8zsr\" (UniqueName: \"kubernetes.io/projected/81621736-37f1-4a1c-be17-6878793b7078-kube-api-access-d8zsr\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:36:25.275231 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.275170    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81621736-37f1-4a1c-be17-6878793b7078-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:36:25.275231 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.275180    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81621736-37f1-4a1c-be17-6878793b7078-proxy-tls\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:36:25.275231 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.275190    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81621736-37f1-4a1c-be17-6878793b7078-cabundle-cert\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:36:25.275231 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.275198    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81621736-37f1-4a1c-be17-6878793b7078-kserve-provision-location\") on node \"ip-10-0-135-44.ec2.internal\" DevicePath \"\""
Sep 16 19:36:25.620182 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.620079    2565 generic.go:356] "Generic (PLEG): container finished" podID="81621736-37f1-4a1c-be17-6878793b7078" containerID="3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950" exitCode=0
Sep 16 19:36:25.620182 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.620161    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"
Sep 16 19:36:25.620383 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.620171    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" event={"ID":"81621736-37f1-4a1c-be17-6878793b7078","Type":"ContainerDied","Data":"3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950"}
Sep 16 19:36:25.620383 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.620250    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw" event={"ID":"81621736-37f1-4a1c-be17-6878793b7078","Type":"ContainerDied","Data":"f069910f7abad32fa80c204443ff35dea1da002bdf6a4f1bf9802c51c5b09556"}
Sep 16 19:36:25.620383 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.620290    2565 scope.go:117] "RemoveContainer" containerID="805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f"
Sep 16 19:36:25.629099 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.629078    2565 scope.go:117] "RemoveContainer" containerID="3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950"
Sep 16 19:36:25.636653 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.636632    2565 scope.go:117] "RemoveContainer" containerID="3e49f5931dd297e425e2414b00cfbfbe72ceac39fbf06ae9843e7eed5c57c52d"
Sep 16 19:36:25.642840 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.642816    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"]
Sep 16 19:36:25.644124 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.644098    2565 scope.go:117] "RemoveContainer" containerID="805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f"
Sep 16 19:36:25.644431 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:36:25.644411    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f\": container with ID starting with 805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f not found: ID does not exist" containerID="805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f"
Sep 16 19:36:25.644507 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.644446    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f"} err="failed to get container status \"805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f\": rpc error: code = NotFound desc = could not find container \"805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f\": container with ID starting with 805bfc744e30eed6366fdc4534aeffac768bf118ed68c6b68357f70a61ff149f not found: ID does not exist"
Sep 16 19:36:25.644507 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.644474    2565 scope.go:117] "RemoveContainer" containerID="3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950"
Sep 16 19:36:25.644758 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:36:25.644740    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950\": container with ID starting with 3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950 not found: ID does not exist" containerID="3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950"
Sep 16 19:36:25.644799 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.644765    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950"} err="failed to get container status \"3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950\": rpc error: code = NotFound desc = could not find container \"3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950\": container with ID starting with 3677db6b11a96cd32245b246b3afb5b599c8355172bbf3c44015c8ef21e76950 not found: ID does not exist"
Sep 16 19:36:25.644799 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.644782    2565 scope.go:117] "RemoveContainer" containerID="3e49f5931dd297e425e2414b00cfbfbe72ceac39fbf06ae9843e7eed5c57c52d"
Sep 16 19:36:25.645058 ip-10-0-135-44 kubenswrapper[2565]: E0916 19:36:25.645041    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e49f5931dd297e425e2414b00cfbfbe72ceac39fbf06ae9843e7eed5c57c52d\": container with ID starting with 3e49f5931dd297e425e2414b00cfbfbe72ceac39fbf06ae9843e7eed5c57c52d not found: ID does not exist" containerID="3e49f5931dd297e425e2414b00cfbfbe72ceac39fbf06ae9843e7eed5c57c52d"
Sep 16 19:36:25.645130 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.645063    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e49f5931dd297e425e2414b00cfbfbe72ceac39fbf06ae9843e7eed5c57c52d"} err="failed to get container status \"3e49f5931dd297e425e2414b00cfbfbe72ceac39fbf06ae9843e7eed5c57c52d\": rpc error: code = NotFound desc = could not find container \"3e49f5931dd297e425e2414b00cfbfbe72ceac39fbf06ae9843e7eed5c57c52d\": container with ID starting with 3e49f5931dd297e425e2414b00cfbfbe72ceac39fbf06ae9843e7eed5c57c52d not found: ID does not exist"
Sep 16 19:36:25.649705 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:25.649682    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1c9c8b-predictor-745dbb8587-46drw"]
Sep 16 19:36:26.751069 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:36:26.751025    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81621736-37f1-4a1c-be17-6878793b7078" path="/var/lib/kubelet/pods/81621736-37f1-4a1c-be17-6878793b7078/volumes"
Sep 16 19:37:52.215473 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:37:52.215390    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6kw62_28a43599-1a6c-4f0e-80a9-032b45196140/global-pull-secret-syncer/0.log"
Sep 16 19:37:52.416559 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:37:52.416525    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j58qd_6d00f6df-66e0-472a-abfc-9fd5e349ef69/konnectivity-agent/0.log"
Sep 16 19:37:52.436765 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:37:52.436734    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-44.ec2.internal_4a0cc900755ac57a20096a2c2cbaeaf7/haproxy/0.log"
Sep 16 19:37:59.092204 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:37:59.092163    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v4mwx_aa4b2ef6-bca2-4442-b89c-d4236b49c2db/node-exporter/0.log"
Sep 16 19:37:59.118156 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:37:59.118131    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v4mwx_aa4b2ef6-bca2-4442-b89c-d4236b49c2db/kube-rbac-proxy/0.log"
Sep 16 19:37:59.141955 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:37:59.141931    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v4mwx_aa4b2ef6-bca2-4442-b89c-d4236b49c2db/init-textfile/0.log"
Sep 16 19:38:02.586172 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586135    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"]
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586391    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b741a3d6-5728-4861-bd3e-c90ef033b4b7" containerName="storage-initializer"
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586402    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b741a3d6-5728-4861-bd3e-c90ef033b4b7" containerName="storage-initializer"
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586412    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container"
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586418    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container"
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586428    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kube-rbac-proxy"
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586455    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kube-rbac-proxy"
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586468    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b741a3d6-5728-4861-bd3e-c90ef033b4b7" containerName="storage-initializer"
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586473    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b741a3d6-5728-4861-bd3e-c90ef033b4b7" containerName="storage-initializer"
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586481    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="storage-initializer"
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586486    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="storage-initializer"
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586527    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b741a3d6-5728-4861-bd3e-c90ef033b4b7" containerName="storage-initializer"
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586535    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kube-rbac-proxy"
Sep 16 19:38:02.586556 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.586542    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81621736-37f1-4a1c-be17-6878793b7078" containerName="kserve-container"
Sep 16 19:38:02.589232 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.589213    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.591980 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.591958    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wwtss\"/\"openshift-service-ca.crt\""
Sep 16 19:38:02.593046 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.593029    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wwtss\"/\"kube-root-ca.crt\""
Sep 16 19:38:02.593137 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.593039    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wwtss\"/\"default-dockercfg-227rs\""
Sep 16 19:38:02.598983 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.598961    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"]
Sep 16 19:38:02.720877 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.720815    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-lib-modules\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.720877 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.720883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-proc\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.721191 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.720944    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-sys\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.721191 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.720995    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-podres\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.721191 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.721035    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57qvv\" (UniqueName: \"kubernetes.io/projected/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-kube-api-access-57qvv\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.821740 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.821683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-lib-modules\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.821958 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.821782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-proc\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.821958 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.821814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-sys\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.821958 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.821843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-podres\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.821958 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.821883    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-lib-modules\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.821958 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.821890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57qvv\" (UniqueName: \"kubernetes.io/projected/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-kube-api-access-57qvv\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.821958 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.821937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-proc\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.822321 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.821961    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-sys\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.822321 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.822003    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-podres\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.831341 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.831305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57qvv\" (UniqueName: \"kubernetes.io/projected/78b74e08-63fa-4e5a-a03b-c2ec6a39bf64-kube-api-access-57qvv\") pod \"perf-node-gather-daemonset-mn5rt\" (UID: \"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:02.899760 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:02.899658    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:03.176781 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:03.176532    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"]
Sep 16 19:38:03.179965 ip-10-0-135-44 kubenswrapper[2565]: W0916 19:38:03.179928    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod78b74e08_63fa_4e5a_a03b_c2ec6a39bf64.slice/crio-dcf09bcd3b3b1bd09ecec7829c8b065d2ac83940d4a404139bfc91fb98bf6887 WatchSource:0}: Error finding container dcf09bcd3b3b1bd09ecec7829c8b065d2ac83940d4a404139bfc91fb98bf6887: Status 404 returned error can't find the container with id dcf09bcd3b3b1bd09ecec7829c8b065d2ac83940d4a404139bfc91fb98bf6887
Sep 16 19:38:03.729150 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:03.729120    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fxx45_917d490d-2d03-4c46-90a4-f31d97a5bc21/dns/0.log"
Sep 16 19:38:03.750357 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:03.750326    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fxx45_917d490d-2d03-4c46-90a4-f31d97a5bc21/kube-rbac-proxy/0.log"
Sep 16 19:38:03.826834 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:03.826797    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9n9vk_8aa789e9-6a45-4199-8137-895a502baed7/dns-node-resolver/0.log"
Sep 16 19:38:03.980248 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:03.980157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt" event={"ID":"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64","Type":"ContainerStarted","Data":"3901d9f6a3e3d92bf4aa5f222ca95fa6494b76579151cc4da7d3a0fbed03f747"}
Sep 16 19:38:03.980248 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:03.980207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt" event={"ID":"78b74e08-63fa-4e5a-a03b-c2ec6a39bf64","Type":"ContainerStarted","Data":"dcf09bcd3b3b1bd09ecec7829c8b065d2ac83940d4a404139bfc91fb98bf6887"}
Sep 16 19:38:03.980440 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:03.980274    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:03.995773 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:03.995728    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt" podStartSLOduration=1.9957140039999999 podStartE2EDuration="1.995714004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:38:03.99487409 +0000 UTC m=+3167.808462951" watchObservedRunningTime="2026-09-16 19:38:03.995714004 +0000 UTC m=+3167.809302863"
Sep 16 19:38:04.401070 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:04.400992    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-phj72_9ae17fba-9a68-4849-a1d4-3c930a0daa61/node-ca/0.log"
Sep 16 19:38:05.604719 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:05.604689    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2w45q_220b1df3-d68f-44be-8bb9-ffcbcf174803/serve-healthcheck-canary/0.log"
Sep 16 19:38:06.162807 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:06.162777    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rjwhv_29016bee-c5d1-4e37-a13d-bce6b454ae8d/kube-rbac-proxy/0.log"
Sep 16 19:38:06.184458 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:06.184434    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rjwhv_29016bee-c5d1-4e37-a13d-bce6b454ae8d/exporter/0.log"
Sep 16 19:38:06.204536 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:06.204505    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rjwhv_29016bee-c5d1-4e37-a13d-bce6b454ae8d/extractor/0.log"
Sep 16 19:38:08.710099 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:08.710060    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-s6lkw_8c19469e-0474-4f15-9d5d-0b3231fc283d/manager/0.log"
Sep 16 19:38:09.183613 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:09.183527    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-m27t7_4af9067d-2ae7-4578-bb13-04e69a21e329/seaweedfs-tls-custom/0.log"
Sep 16 19:38:09.207222 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:09.207186    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-2vwxr_1b4495f8-ff25-4061-b0df-857669caf6b0/seaweedfs-tls-serving/0.log"
Sep 16 19:38:09.995586 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:09.995558    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-mn5rt"
Sep 16 19:38:15.032284 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:15.032250    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7kqt2_4ad64d2a-fd0c-45f9-be97-119db12f0292/kube-multus/0.log"
Sep 16 19:38:15.053627 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:15.053600    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kq9p_023ad12c-f4a4-43cf-b91a-7004d2eff4ed/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:38:15.078852 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:15.078827    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kq9p_023ad12c-f4a4-43cf-b91a-7004d2eff4ed/egress-router-binary-copy/0.log"
Sep 16 19:38:15.101705 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:15.101685    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kq9p_023ad12c-f4a4-43cf-b91a-7004d2eff4ed/cni-plugins/0.log"
Sep 16 19:38:15.123260 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:15.123243    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kq9p_023ad12c-f4a4-43cf-b91a-7004d2eff4ed/bond-cni-plugin/0.log"
Sep 16 19:38:15.148436 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:15.148415    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kq9p_023ad12c-f4a4-43cf-b91a-7004d2eff4ed/routeoverride-cni/0.log"
Sep 16 19:38:15.171729 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:15.171661    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kq9p_023ad12c-f4a4-43cf-b91a-7004d2eff4ed/whereabouts-cni-bincopy/0.log"
Sep 16 19:38:15.195445 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:15.195423    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kq9p_023ad12c-f4a4-43cf-b91a-7004d2eff4ed/whereabouts-cni/0.log"
Sep 16 19:38:15.565305 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:15.565274    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qz4s2_5dda191a-363f-44c4-9f13-d1f59108abbc/network-metrics-daemon/0.log"
Sep 16 19:38:15.586140 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:15.586118    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qz4s2_5dda191a-363f-44c4-9f13-d1f59108abbc/kube-rbac-proxy/0.log"
Sep 16 19:38:16.888168 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:16.888141    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thbb5_686c60bb-f175-4c94-9b73-129291c971b4/ovn-controller/0.log"
Sep 16 19:38:16.905288 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:16.905268    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thbb5_686c60bb-f175-4c94-9b73-129291c971b4/ovn-acl-logging/0.log"
Sep 16 19:38:16.928949 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:16.928928    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thbb5_686c60bb-f175-4c94-9b73-129291c971b4/kube-rbac-proxy-node/0.log"
Sep 16 19:38:16.951639 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:16.951613    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thbb5_686c60bb-f175-4c94-9b73-129291c971b4/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:38:16.971880 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:16.971856    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thbb5_686c60bb-f175-4c94-9b73-129291c971b4/northd/0.log"
Sep 16 19:38:16.996282 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:16.996259    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thbb5_686c60bb-f175-4c94-9b73-129291c971b4/nbdb/0.log"
Sep 16 19:38:17.020573 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:17.020553    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thbb5_686c60bb-f175-4c94-9b73-129291c971b4/sbdb/0.log"
Sep 16 19:38:17.080951 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:17.080927    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thbb5_686c60bb-f175-4c94-9b73-129291c971b4/ovnkube-controller/0.log"
Sep 16 19:38:18.153320 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:18.153288    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-lfqsq_9f5c39f3-f783-4d49-b3f1-3cd4392d347c/network-check-target-container/0.log"
Sep 16 19:38:19.130982 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:19.130954    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-j88k9_66a5cb54-a1df-40ca-a945-a5378898cf93/iptables-alerter/0.log"
Sep 16 19:38:19.860697 ip-10-0-135-44 kubenswrapper[2565]: I0916 19:38:19.860669    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-5qg59_1bf127e9-9ab0-4581-a467-62bdaa8052f5/tuned/0.log"