Jul 20 09:56:58.584194 ip-10-0-134-80 systemd[1]: Starting Kubernetes Kubelet...
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.040121    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044559    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044570    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044573    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044576    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044579    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044582    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044586    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:59.138999 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044589    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:59.122979 ip-10-0-134-80 systemd[1]: Started Kubernetes Kubelet.
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044592    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044595    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044598    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044601    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044603    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044612    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044615    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044618    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044620    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044623    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044630    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044633    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044635    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044639    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044643    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044647    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044650    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044654    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044656    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:59.148654 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044659    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044662    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044664    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044667    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044670    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044673    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044677    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044680    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044683    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044685    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044688    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044690    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044693    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044704    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044707    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044710    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044713    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044716    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044719    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:59.152751 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044722    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044724    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044727    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044729    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044731    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044734    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044737    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044739    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044741    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044745    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044747    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044750    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044753    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044755    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044758    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044760    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044763    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044766    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044768    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:59.155118 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044771    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044773    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044776    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044778    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044780    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044783    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044785    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044788    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044796    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044799    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044801    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044803    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044806    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044809    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044811    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044815    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044817    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044820    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044822    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044824    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:59.159684 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044827    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044842    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044846    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044849    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044852    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044855    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044857    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044859    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044862    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044865    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044867    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044870    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044872    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.044875    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048181    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048190    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048193    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048196    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048199    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048203    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:59.162023 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048207    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048211    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048214    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048217    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048220    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048223    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048225    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048228    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048231    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048233    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048236    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048238    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048241    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048243    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048246    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048249    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048252    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048255    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048257    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048260    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:59.164383 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048263    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048265    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048268    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048270    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048273    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048276    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048279    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048281    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048284    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048286    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048289    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048291    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048294    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048296    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048305    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048308    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048311    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048314    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048318    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048321    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:59.165425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048324    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048327    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048329    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048332    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048334    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048337    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048340    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048343    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048346    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048348    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048351    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048353    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048356    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048358    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048361    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048364    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048366    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048369    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048371    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048374    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:59.166917 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048376    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048379    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048381    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048385    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048387    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048389    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048392    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048394    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048397    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048399    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048402    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048404    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048407    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048409    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048411    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048414    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048416    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048418    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048421    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:59.169299 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048424    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048426    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048429    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048431    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048435    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048438    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048441    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048444    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048446    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048449    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048452    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048454    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048457    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048460    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048804    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048809    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048811    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048814    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048817    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048819    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:59.170060 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048822    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048825    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048827    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048845    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048850    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048853    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048856    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048859    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048862    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048864    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048867    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048870    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048873    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048876    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048879    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048882    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048885    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048890    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048893    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:59.170793 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048896    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048899    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048902    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048904    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048907    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048909    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048912    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048914    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048917    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048922    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048924    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048927    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048929    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048932    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048935    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048937    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048940    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048942    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048945    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048947    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:59.171501 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048950    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048952    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048955    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048957    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048963    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048965    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048968    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048971    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048973    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048976    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048978    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048980    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048983    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048986    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048988    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048992    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048995    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.048998    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049001    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:59.172256 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049004    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049007    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049011    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049014    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049017    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049020    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049023    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049025    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049028    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049031    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049033    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049036    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049038    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049041    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049043    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049046    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049048    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049052    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049055    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:59.172945 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049059    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049061    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049064    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049066    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049069    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049071    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049074    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049076    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049078    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049081    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049084    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049086    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049088    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049091    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049093    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049097    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049173    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049180    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049187    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049191    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049195    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 20 09:56:59.173662 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049199    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049203    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049207    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049210    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049212    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049216    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049219    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049222    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049225    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049229    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049231    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049235    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049238    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049242    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049244    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049247    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049250    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049254    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049257    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049260    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049263    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049266    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049269    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049272    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 20 09:56:59.174479 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049275    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049278    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049281    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049290    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049293    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049296    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049298    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049301    2561 flags.go:64] FLAG: --enable-server="true"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049304    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049311    2561 flags.go:64] FLAG: --event-burst="100"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049315    2561 flags.go:64] FLAG: --event-qps="50"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049318    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049321    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049324    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049327    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049330    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049333    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049337    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049340    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049342    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049345    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049348    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049351    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049354    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049357    2561 flags.go:64] FLAG: --feature-gates=""
Jul 20 09:56:59.175352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049360    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049363    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049366    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049369    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049372    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049375    2561 flags.go:64] FLAG: --help="false"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049378    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-134-80.ec2.internal"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049381    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049384    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049388    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049391    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049394    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049397    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049400    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049403    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049406    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049408    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049411    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049415    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049418    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049420    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049423    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049426    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049429    2561 flags.go:64] FLAG: --lock-file=""
Jul 20 09:56:59.176322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049433    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049436    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049439    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049444    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049447    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049450    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049452    2561 flags.go:64] FLAG: --logging-format="text"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049455    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049459    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049461    2561 flags.go:64] FLAG: --manifest-url=""
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049464    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049468    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049471    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049475    2561 flags.go:64] FLAG: --max-pods="110"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049478    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049481    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049484    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049488    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049491    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049493    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049496    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049503    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049506    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049509    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 20 09:56:59.177245 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049512    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049514    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049520    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049523    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049526    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049529    2561 flags.go:64] FLAG: --port="10250"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049532    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049535    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07d97c0993f919910"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049538    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049542    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049545    2561 flags.go:64] FLAG: --register-node="true"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049548    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049551    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049554    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049557    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049560    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049564    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049567    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049570    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049572    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049575    2561 flags.go:64] FLAG: --runonce="false"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049578    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049581    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049584    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049587    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 20 09:56:59.178145 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049591    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049594    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049597    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049600    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049603    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049606    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049608    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049611    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049614    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049617    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049620    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049625    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049628    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049631    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049635    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049638    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049642    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049644    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049647    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049650    2561 flags.go:64] FLAG: --v="2"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049654    2561 flags.go:64] FLAG: --version="false"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049658    2561 flags.go:64] FLAG: --vmodule=""
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049661    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.049664    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049769    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:59.179084 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049773    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049776    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049779    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049781    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049784    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049787    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049789    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049794    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049796    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049799    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049801    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049804    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049807    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049809    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049812    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049815    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049818    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049820    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049823    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049827    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:59.180081 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049845    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049848    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049851    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049855    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049857    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049860    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049863    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049865    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049868    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049870    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049873    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049876    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049878    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049881    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049883    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049886    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049888    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049891    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049893    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049897    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:59.180851 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049900    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049902    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049905    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049908    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049910    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049914    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049917    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049921    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049923    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049926    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049929    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049932    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049935    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049938    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049940    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049944    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049947    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049950    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049952    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:59.181606 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049955    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049958    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049960    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049963    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049965    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049968    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049971    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049973    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049976    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049979    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049981    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049984    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049988    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049991    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049993    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049996    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.049999    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050001    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050004    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050006    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:59.182342 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050009    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050011    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050015    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050018    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050021    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050024    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050028    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050030    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050034    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050037    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050039    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050042    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050045    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050047    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050050    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050053    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050055    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050058    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:59.183114 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.050060    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.050763    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.058742    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.058758    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.058777    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.058783    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058855    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058860    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058864    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058867    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058870    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058873    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058876    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058879    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058882    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:59.183796 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058885    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058887    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058890    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058893    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058896    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058899    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058901    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058903    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058906    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058910    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058914    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058917    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058920    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058922    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058925    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058928    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058931    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058934    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058937    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:59.184373 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058940    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058944    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058948    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058952    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058955    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058957    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058960    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058962    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058965    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058969    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058972    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058976    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058979    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058982    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058984    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058987    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058989    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058992    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058994    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058997    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:59.185086 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.058999    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059002    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059004    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059007    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059009    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059012    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059015    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059017    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059020    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059022    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059025    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059028    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059030    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059034    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059036    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059039    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059041    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059044    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059047    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:59.185829 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059049    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059052    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059054    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059057    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059060    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059062    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059065    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059067    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059070    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059072    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059075    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059077    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059080    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059083    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059086    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059088    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059094    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059096    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059099    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059101    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:59.186543 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059105    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059107    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059110    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059112    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059115    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059118    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059120    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059123    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059126    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059128    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059131    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059133    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.059138    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059265    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059270    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:59.187291 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059273    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059276    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059279    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059282    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059285    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059288    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059291    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059294    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059297    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059300    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059304    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059307    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059309    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059312    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059315    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059318    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059321    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059323    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059326    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059328    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:59.187874 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059331    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059334    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059338    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059340    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059342    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059345    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059348    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059350    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059352    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059355    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059358    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059360    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059363    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059365    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059368    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059371    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059373    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059376    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059379    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059381    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:59.188621 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059383    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059386    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059388    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059391    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059394    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059396    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059400    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059402    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059405    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059407    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059411    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059415    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059418    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059421    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059426    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059429    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059432    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059435    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059438    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:59.189385 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059440    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059443    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059446    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059448    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059451    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059454    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059456    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059459    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059461    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059464    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059466    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059469    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059471    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059474    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059476    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059479    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059481    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059485    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059487    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059492    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:59.190096 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059495    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059499    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059501    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059504    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059507    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059510    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059512    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059516    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059519    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059522    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059525    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059528    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059531    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059534    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059537    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059539    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059542    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:59.190842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:56:59.059544    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:59.191484 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.059549    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:59.191484 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.059640    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 20 09:56:59.191484 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.062746    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 20 09:56:59.191484 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.064292    2561 server.go:1019] "Starting client certificate rotation"
Jul 20 09:56:59.191484 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.064388    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 09:56:59.191484 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.064421    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 09:56:59.191484 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.067256    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 09:56:59.191484 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.072800    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 09:56:59.191484 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.088063    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 20 09:56:59.191484 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.090571    2561 log.go:25] "Validated CRI v1 image API"
Jul 20 09:56:59.191484 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.091437    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 20 09:56:59.191484 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.095518    2561 fs.go:135] Filesystem UUIDs: map[5abcaf88-8402-4675-95e4-ba1cbf3156b9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bce9b997-a0ed-4c44-ad86-674e37ffa6fe:/dev/nvme0n1p4]
Jul 20 09:56:59.191959 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.095531    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 20 09:56:59.192007 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.101183    2561 manager.go:216] Machine: {Timestamp:2026-07-20 09:56:59.09984778 +0000 UTC m=+0.390225855 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096136 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec227f0203826666962477558c9ddbd1 SystemUUID:ec227f02-0382-6666-9624-77558c9ddbd1 BootID:c6eb1c87-1044-4bf7-97d3-0d3fa53cebdc Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7c:7e:a1:a7:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7c:7e:a1:a7:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:19:02:b7:33:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 20 09:56:59.192007 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.101283    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 20 09:56:59.192007 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.101359    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 20 09:56:59.192007 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.102497    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 20 09:56:59.192007 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.102521    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 20 09:56:59.192007 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.103368    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 20 09:56:59.192007 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.103385    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 20 09:56:59.192007 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.103392    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.103413    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.105197    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.105233    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.109177    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.109302    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.109329    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.109339    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.109352    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.109361    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.109369    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.109736    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.109756    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.113118    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.116925    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.116951    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.118179    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.118192    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.118198    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.118203    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.118212    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.118221    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.118233    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.118241    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.118248    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.118253    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.118261    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.118266    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.119044    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.119055    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.120212    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.120225    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.122151    2561 server.go:1268] "Started kubelet"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.122216    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.123007    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.123082    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.124180    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.124301    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.128629    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 20 09:56:59.192362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.134786    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.134352    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-80.ec2.internal.18c3f6f73dbed438  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-80.ec2.internal,UID:ip-10-0-134-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-80.ec2.internal,},FirstTimestamp:2026-07-20 09:56:59.122119736 +0000 UTC m=+0.412497792,LastTimestamp:2026-07-20 09:56:59.122119736 +0000 UTC m=+0.412497792,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-80.ec2.internal,}"
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.135578    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.135748    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.136163    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.136855    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.136888    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.136905    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.136986    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.137028    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.137270    2561 factory.go:55] Registering systemd factory
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.137285    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.137521    2561 factory.go:153] Registering CRI-O factory
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.137537    2561 factory.go:223] Registration of the crio container factory successfully
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.137587    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.137607    2561 factory.go:103] Registering Raw factory
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.137621    2561 manager.go:1196] Started watching for new ooms in manager
Jul 20 09:56:59.193920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.138138    2561 manager.go:349] Starting recovery of all containers
Jul 20 09:56:59.210891 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.210762    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qzmwz"
Jul 20 09:56:59.210891 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.210891    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qzmwz"
Jul 20 09:56:59.215545 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.215478    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 20 09:56:59.215723 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.215693    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 20 09:56:59.221647 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.221628    2561 manager.go:354] Recovery completed
Jul 20 09:56:59.225953 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.225941    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:59.228819 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.228796    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:59.228913 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.228828    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:59.228913 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.228854    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:59.229410 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.229398    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 20 09:56:59.229410 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.229408    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 20 09:56:59.229503 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.229418    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:59.229503 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.229422    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 09:56:59.230385 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.230373    2561 policy_none.go:49] "None policy: Start"
Jul 20 09:56:59.230448 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.230388    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 20 09:56:59.230448 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.230398    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 20 09:56:59.230874 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.230757    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-80.ec2.internal.18c3f6f7441adbc0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-80.ec2.internal,UID:ip-10-0-134-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-80.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-80.ec2.internal,},FirstTimestamp:2026-07-20 09:56:59.228814272 +0000 UTC m=+0.519192331,LastTimestamp:2026-07-20 09:56:59.228814272 +0000 UTC m=+0.519192331,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-80.ec2.internal,}"
Jul 20 09:56:59.230964 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.230933    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:59.230964 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.230957    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:59.231039 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.230967    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:59.231257 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.231246    2561 policy_none.go:47] "Start"
Jul 20 09:56:59.236587 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.236426    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:56:59.266439 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.266427    2561 manager.go:337] "Starting Device Plugin manager"
Jul 20 09:56:59.266526 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.266455    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 20 09:56:59.266526 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.266466    2561 server.go:87] "Starting device plugin registration server"
Jul 20 09:56:59.266717 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.266706    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 20 09:56:59.266768 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.266721    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 20 09:56:59.266919 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.266904    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 20 09:56:59.267010 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.266995    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 20 09:56:59.267010 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.267005    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 20 09:56:59.267478 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.267445    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 20 09:56:59.267555 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.267513    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:56:59.306697 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.306676    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 20 09:56:59.307752 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.307737    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 20 09:56:59.307826 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.307779    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 20 09:56:59.307826 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.307802    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 20 09:56:59.307944 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.307851    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 20 09:56:59.310548 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.310534    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:59.367421 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.367404    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:59.368223 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.368205    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:59.368294 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.368237    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:59.368294 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.368248    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:59.368294 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.368273    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-80.ec2.internal"
Jul 20 09:56:59.382763 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.382748    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-80.ec2.internal"
Jul 20 09:56:59.382821 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.382765    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-80.ec2.internal\": node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:56:59.404096 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.404055    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:56:59.408129 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.408114    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-80.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal"]
Jul 20 09:56:59.408184 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.408174    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:59.409470 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.409452    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:59.409549 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.409479    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:59.409549 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.409494    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:59.410588 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.410577    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:59.410681 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.410669    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-80.ec2.internal"
Jul 20 09:56:59.410716 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.410697    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:59.413912 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.413898    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:59.413985 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.413923    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:59.413985 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.413932    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:59.413985 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.413903    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:59.414086 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.413994    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:59.414086 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.414003    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:59.415125 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.415112    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal"
Jul 20 09:56:59.415173 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.415135    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:59.415907 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.415893    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:59.415972 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.415923    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:59.415972 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.415940    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:59.438392 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.438365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8acf0f9821033101209753c1ff5ad61-config\") pod \"kube-apiserver-proxy-ip-10-0-134-80.ec2.internal\" (UID: \"d8acf0f9821033101209753c1ff5ad61\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-80.ec2.internal"
Jul 20 09:56:59.438473 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.438394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00c0661b8872f25c32c9ae0016e0b8ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal\" (UID: \"00c0661b8872f25c32c9ae0016e0b8ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal"
Jul 20 09:56:59.438473 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.438410    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-80.ec2.internal\" not found" node="ip-10-0-134-80.ec2.internal"
Jul 20 09:56:59.438473 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:56:59.438416    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00c0661b8872f25c32c9ae0016e0b8ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal\" (UID: \"00c0661b8872f25c32c9ae0016e0b8ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal"
Jul 20 09:56:59.441911 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.441897    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-80.ec2.internal\" not found" node="ip-10-0-134-80.ec2.internal"
Jul 20 09:56:59.504843 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:56:59.504816    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:57:00.991586 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:00.991568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00c0661b8872f25c32c9ae0016e0b8ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal\" (UID: \"00c0661b8872f25c32c9ae0016e0b8ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal"
Jul 20 09:57:00.991649 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:00.991589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00c0661b8872f25c32c9ae0016e0b8ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal\" (UID: \"00c0661b8872f25c32c9ae0016e0b8ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal"
Jul 20 09:57:00.991649 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:00.991621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8acf0f9821033101209753c1ff5ad61-config\") pod \"kube-apiserver-proxy-ip-10-0-134-80.ec2.internal\" (UID: \"d8acf0f9821033101209753c1ff5ad61\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-80.ec2.internal"
Jul 20 09:57:00.991722 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:00.991677    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8acf0f9821033101209753c1ff5ad61-config\") pod \"kube-apiserver-proxy-ip-10-0-134-80.ec2.internal\" (UID: \"d8acf0f9821033101209753c1ff5ad61\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-80.ec2.internal"
Jul 20 09:57:00.991722 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:00.991718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00c0661b8872f25c32c9ae0016e0b8ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal\" (UID: \"00c0661b8872f25c32c9ae0016e0b8ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal"
Jul 20 09:57:00.991792 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:00.991751    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00c0661b8872f25c32c9ae0016e0b8ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal\" (UID: \"00c0661b8872f25c32c9ae0016e0b8ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal"
Jul 20 09:57:01.057367 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:01.057347    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:57:01.158190 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:01.158127    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:57:01.194952 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:01.194932    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-80.ec2.internal"
Jul 20 09:57:01.197039 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:01.197025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal"
Jul 20 09:57:01.259009 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:01.258975    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:57:01.359525 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:01.359499    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:57:01.460040 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:01.459980    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:57:01.516511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:01.516493    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 20 09:57:01.516641 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:01.516624    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 09:57:01.560805 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:01.560782    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:57:01.661811 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:01.661791    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:57:01.668955 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:01.668929    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 09:52:00 +0000 UTC" deadline="2028-01-08 17:40:36.409449456 +0000 UTC"
Jul 20 09:57:01.669028 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:01.668955    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12895h43m34.740498836s"
Jul 20 09:57:01.753957 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:01.753918    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c0661b8872f25c32c9ae0016e0b8ef.slice/crio-2e3adfe0fe406567a2beee666ef74adb68acef467c5b398703ba1724ce6596e8 WatchSource:0}: Error finding container 2e3adfe0fe406567a2beee666ef74adb68acef467c5b398703ba1724ce6596e8: Status 404 returned error can't find the container with id 2e3adfe0fe406567a2beee666ef74adb68acef467c5b398703ba1724ce6596e8
Jul 20 09:57:01.754233 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:01.754203    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8acf0f9821033101209753c1ff5ad61.slice/crio-600aecdb70cd6af2e0a003c8f5f2d2c31c12597660ecf27318553eb7e6228379 WatchSource:0}: Error finding container 600aecdb70cd6af2e0a003c8f5f2d2c31c12597660ecf27318553eb7e6228379: Status 404 returned error can't find the container with id 600aecdb70cd6af2e0a003c8f5f2d2c31c12597660ecf27318553eb7e6228379
Jul 20 09:57:01.759107 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:01.759084    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 09:57:01.761848 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:01.761831    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:57:01.763443 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:01.763394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-80.ec2.internal" event={"ID":"d8acf0f9821033101209753c1ff5ad61","Type":"ContainerStarted","Data":"600aecdb70cd6af2e0a003c8f5f2d2c31c12597660ecf27318553eb7e6228379"}
Jul 20 09:57:01.764462 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:01.764444    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal" event={"ID":"00c0661b8872f25c32c9ae0016e0b8ef","Type":"ContainerStarted","Data":"2e3adfe0fe406567a2beee666ef74adb68acef467c5b398703ba1724ce6596e8"}
Jul 20 09:57:01.862469 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:01.862444    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:57:01.962971 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:01.962940    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-80.ec2.internal\" not found"
Jul 20 09:57:01.978443 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:01.978423    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:02.032109 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.032052    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:02.039791 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.039776    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:02.089143 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.089124    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-80.ec2.internal"
Jul 20 09:57:02.100009 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.099995    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 09:57:02.100985 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.100973    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal"
Jul 20 09:57:02.116730 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.116712    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 09:57:02.562700 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.562672    2561 apiserver.go:52] "Watching apiserver"
Jul 20 09:57:02.572271 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.572247    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 20 09:57:02.572799 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.572770    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rw9rr","kube-system/konnectivity-agent-kj6v2","kube-system/kube-apiserver-proxy-ip-10-0-134-80.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qsb9p","openshift-dns/node-resolver-mn8bv","openshift-image-registry/node-ca-n5hmv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal","openshift-multus/multus-l8xwg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw","openshift-multus/multus-additional-cni-plugins-n8s76","openshift-multus/network-metrics-daemon-n4x2n","openshift-network-diagnostics/network-check-target-fcpp5","openshift-network-operator/iptables-alerter-2wmxz","openshift-ovn-kubernetes/ovnkube-node-kxc68"]
Jul 20 09:57:02.574316 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.574292    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.575947 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.575493    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.576842 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.576763    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:02.576939 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:02.576837    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:02.577905 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.577885    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.579511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.579493    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.580134 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.580101    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 20 09:57:02.580206 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.580149    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.580360 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.580344    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 20 09:57:02.580493 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.580475    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.580553 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.580477    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n5hmv"
Jul 20 09:57:02.582103 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.581682    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mn8bv"
Jul 20 09:57:02.582981 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.582963    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.584499 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.584479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:02.584588 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:02.584540    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:02.586095 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.586077    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:02.586208 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:02.586147    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:02.587571 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.586597    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 20 09:57:02.587571 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.586911    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v7mmm\""
Jul 20 09:57:02.587571 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.587014    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.587571 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.587102    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.587571 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.587299    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.587571 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.587335    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.587964 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.587773    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2wmxz"
Jul 20 09:57:02.588798 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.588388    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.588798 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.588458    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.588798 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.588611    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 20 09:57:02.589006 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.588991    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kj6v2"
Jul 20 09:57:02.590447 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.590359    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 20 09:57:02.590447 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.590413    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 20 09:57:02.590605 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.590574    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 20 09:57:02.590780 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.590758    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 20 09:57:02.591035 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.591018    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.591295 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.591197    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.591960 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.591697    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.596221 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.592792    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vf5zw\""
Jul 20 09:57:02.596221 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.591901    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h5cmc\""
Jul 20 09:57:02.596221 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.593131    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 20 09:57:02.596221 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.592797    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7vgkq\""
Jul 20 09:57:02.596221 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.594140    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:02.596221 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.594355    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.596221 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.594635    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 20 09:57:02.596221 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.595232    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 20 09:57:02.596221 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.595442    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:02.596221 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.595835    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dcw9f\""
Jul 20 09:57:02.596221 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.595911    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z4ghz\""
Jul 20 09:57:02.596698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.596273    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9swpg\""
Jul 20 09:57:02.596698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.596383    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 20 09:57:02.596698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.596452    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b669x\""
Jul 20 09:57:02.596698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.596607    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 20 09:57:02.596882 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.596801    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mm4tq\""
Jul 20 09:57:02.597570 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.597035    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 20 09:57:02.597570 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.597051    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 20 09:57:02.600437 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/102e4f9f-8d5c-40d6-8b00-7f1a13888508-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.600537 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600452    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-socket-dir\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.600537 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c54h\" (UniqueName: \"kubernetes.io/projected/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-kube-api-access-4c54h\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.600537 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-sysctl-d\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.600537 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d85828e-89be-41f7-a3e6-9011e4280223-etc-tuned\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.600732 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33f1203a-dd9a-46be-96bd-47184ae3fccf-serviceca\") pod \"node-ca-n5hmv\" (UID: \"33f1203a-dd9a-46be-96bd-47184ae3fccf\") " pod="openshift-image-registry/node-ca-n5hmv"
Jul 20 09:57:02.600732 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ab13120-6b0f-4805-bc85-bb1ee7384987-kubelet-config\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:02.600732 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-etc-selinux\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.600732 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c840916c-ce29-4f39-8f28-b2c462411b72-iptables-alerter-script\") pod \"iptables-alerter-2wmxz\" (UID: \"c840916c-ce29-4f39-8f28-b2c462411b72\") " pod="openshift-network-operator/iptables-alerter-2wmxz"
Jul 20 09:57:02.600732 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-multus-socket-dir-parent\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.600732 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-run-openvswitch\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.600732 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-cni-bin\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.600732 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.601086 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-ovn-node-metrics-cert\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.601086 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/102e4f9f-8d5c-40d6-8b00-7f1a13888508-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.601086 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600891    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-multus-cni-dir\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.601086 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600918    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-ovnkube-script-lib\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.601086 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-sysconfig\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.601086 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-device-dir\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.601086 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.600986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8h5s\" (UniqueName: \"kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s\") pod \"network-check-target-fcpp5\" (UID: \"ffd754c6-58d4-429e-aa73-9e3cd0f1626e\") " pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:02.601086 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-systemd-units\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.601086 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-etc-openvswitch\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-cni-netd\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601139    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-os-release\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-cni-binary-copy\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601186    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-multus-conf-dir\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-slash\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601228    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ab13120-6b0f-4805-bc85-bb1ee7384987-dbus\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601254    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcsdr\" (UniqueName: \"kubernetes.io/projected/102e4f9f-8d5c-40d6-8b00-7f1a13888508-kube-api-access-vcsdr\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-var-lib-cni-bin\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601322    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-systemd\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601345    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-run-k8s-cni-cncf-io\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601413    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-var-lib-cni-multus\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.601522 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601474    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/102e4f9f-8d5c-40d6-8b00-7f1a13888508-system-cni-dir\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-system-cni-dir\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601629    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-run-netns\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601656    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-sys\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33f1203a-dd9a-46be-96bd-47184ae3fccf-host\") pod \"node-ca-n5hmv\" (UID: \"33f1203a-dd9a-46be-96bd-47184ae3fccf\") " pod="openshift-image-registry/node-ca-n5hmv"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-var-lib-kubelet\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-kubelet\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-sysctl-conf\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-host\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.601983    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-kubernetes\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602006    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmx5z\" (UniqueName: \"kubernetes.io/projected/df44767e-e243-4004-a43e-4d2982ddd197-kube-api-access-pmx5z\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602029    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/102e4f9f-8d5c-40d6-8b00-7f1a13888508-cnibin\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602069    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m856h\" (UniqueName: \"kubernetes.io/projected/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-kube-api-access-m856h\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.602191 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-hostroot\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-multus-daemon-config\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602221    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4skps\" (UniqueName: \"kubernetes.io/projected/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-kube-api-access-4skps\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-node-log\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602276    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-log-socket\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602299    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thw9d\" (UniqueName: \"kubernetes.io/projected/c840916c-ce29-4f39-8f28-b2c462411b72-kube-api-access-thw9d\") pod \"iptables-alerter-2wmxz\" (UID: \"c840916c-ce29-4f39-8f28-b2c462411b72\") " pod="openshift-network-operator/iptables-alerter-2wmxz"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602323    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/102e4f9f-8d5c-40d6-8b00-7f1a13888508-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a052b9a-0d1c-4b01-8b75-1f435ec83841-tmp-dir\") pod \"node-resolver-mn8bv\" (UID: \"6a052b9a-0d1c-4b01-8b75-1f435ec83841\") " pod="openshift-dns/node-resolver-mn8bv"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602380    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-sys-fs\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-run-ovn\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602423    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-ovnkube-config\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-env-overrides\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-modprobe-d\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c840916c-ce29-4f39-8f28-b2c462411b72-host-slash\") pod \"iptables-alerter-2wmxz\" (UID: \"c840916c-ce29-4f39-8f28-b2c462411b72\") " pod="openshift-network-operator/iptables-alerter-2wmxz"
Jul 20 09:57:02.602916 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/013cc0d4-fe49-43cd-9fb5-1041e0b0001d-agent-certs\") pod \"konnectivity-agent-kj6v2\" (UID: \"013cc0d4-fe49-43cd-9fb5-1041e0b0001d\") " pod="kube-system/konnectivity-agent-kj6v2"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602562    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/013cc0d4-fe49-43cd-9fb5-1041e0b0001d-konnectivity-ca\") pod \"konnectivity-agent-kj6v2\" (UID: \"013cc0d4-fe49-43cd-9fb5-1041e0b0001d\") " pod="kube-system/konnectivity-agent-kj6v2"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602586    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-run-multus-certs\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602613    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-var-lib-openvswitch\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-var-lib-kubelet\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602656    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d85828e-89be-41f7-a3e6-9011e4280223-tmp\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/102e4f9f-8d5c-40d6-8b00-7f1a13888508-os-release\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/102e4f9f-8d5c-40d6-8b00-7f1a13888508-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602767    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a052b9a-0d1c-4b01-8b75-1f435ec83841-hosts-file\") pod \"node-resolver-mn8bv\" (UID: \"6a052b9a-0d1c-4b01-8b75-1f435ec83841\") " pod="openshift-dns/node-resolver-mn8bv"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s879n\" (UniqueName: \"kubernetes.io/projected/6a052b9a-0d1c-4b01-8b75-1f435ec83841-kube-api-access-s879n\") pod \"node-resolver-mn8bv\" (UID: \"6a052b9a-0d1c-4b01-8b75-1f435ec83841\") " pod="openshift-dns/node-resolver-mn8bv"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-run-netns\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-run\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-cnibin\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.603599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602958    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-lib-modules\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.604173 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.602998    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krnck\" (UniqueName: \"kubernetes.io/projected/33f1203a-dd9a-46be-96bd-47184ae3fccf-kube-api-access-krnck\") pod \"node-ca-n5hmv\" (UID: \"33f1203a-dd9a-46be-96bd-47184ae3fccf\") " pod="openshift-image-registry/node-ca-n5hmv"
Jul 20 09:57:02.604173 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.603025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-registration-dir\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.604173 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.603067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-etc-kubernetes\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.604173 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.603093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-run-systemd\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.604173 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.603135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbggz\" (UniqueName: \"kubernetes.io/projected/8d85828e-89be-41f7-a3e6-9011e4280223-kube-api-access-pbggz\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.606718 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.606699    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 09:57:02.634415 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.634389    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-njh88"
Jul 20 09:57:02.638498 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.638477    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:57:02.642323 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.642301    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-njh88"
Jul 20 09:57:02.703359 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-cnibin\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-lib-modules\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703398    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krnck\" (UniqueName: \"kubernetes.io/projected/33f1203a-dd9a-46be-96bd-47184ae3fccf-kube-api-access-krnck\") pod \"node-ca-n5hmv\" (UID: \"33f1203a-dd9a-46be-96bd-47184ae3fccf\") " pod="openshift-image-registry/node-ca-n5hmv"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-registration-dir\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-etc-kubernetes\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-run-systemd\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbggz\" (UniqueName: \"kubernetes.io/projected/8d85828e-89be-41f7-a3e6-9011e4280223-kube-api-access-pbggz\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/102e4f9f-8d5c-40d6-8b00-7f1a13888508-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-socket-dir\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c54h\" (UniqueName: \"kubernetes.io/projected/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-kube-api-access-4c54h\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-sysctl-d\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d85828e-89be-41f7-a3e6-9011e4280223-etc-tuned\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33f1203a-dd9a-46be-96bd-47184ae3fccf-serviceca\") pod \"node-ca-n5hmv\" (UID: \"33f1203a-dd9a-46be-96bd-47184ae3fccf\") " pod="openshift-image-registry/node-ca-n5hmv"
Jul 20 09:57:02.703793 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703692    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ab13120-6b0f-4805-bc85-bb1ee7384987-kubelet-config\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703833    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-run-systemd\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-registration-dir\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.703953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-etc-kubernetes\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704048    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-sysctl-d\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-lib-modules\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704107    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-cnibin\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704161    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-etc-selinux\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c840916c-ce29-4f39-8f28-b2c462411b72-iptables-alerter-script\") pod \"iptables-alerter-2wmxz\" (UID: \"c840916c-ce29-4f39-8f28-b2c462411b72\") " pod="openshift-network-operator/iptables-alerter-2wmxz"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-socket-dir\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-multus-socket-dir-parent\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-run-openvswitch\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-cni-bin\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.704356 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704300    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-etc-selinux\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-ovn-node-metrics-cert\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/102e4f9f-8d5c-40d6-8b00-7f1a13888508-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-run-openvswitch\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704357    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-multus-cni-dir\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-ovnkube-script-lib\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704411    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-sysconfig\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-device-dir\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h5s\" (UniqueName: \"kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s\") pod \"network-check-target-fcpp5\" (UID: \"ffd754c6-58d4-429e-aa73-9e3cd0f1626e\") " pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-systemd-units\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-etc-openvswitch\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-cni-netd\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704583    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-os-release\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-cni-binary-copy\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.704954 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704628    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-multus-conf-dir\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704646    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ab13120-6b0f-4805-bc85-bb1ee7384987-kubelet-config\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-slash\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ab13120-6b0f-4805-bc85-bb1ee7384987-dbus\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcsdr\" (UniqueName: \"kubernetes.io/projected/102e4f9f-8d5c-40d6-8b00-7f1a13888508-kube-api-access-vcsdr\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704786    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-var-lib-cni-bin\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704832    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-multus-socket-dir-parent\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-cni-bin\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-systemd\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c840916c-ce29-4f39-8f28-b2c462411b72-iptables-alerter-script\") pod \"iptables-alerter-2wmxz\" (UID: \"c840916c-ce29-4f39-8f28-b2c462411b72\") " pod="openshift-network-operator/iptables-alerter-2wmxz"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704958    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-run-k8s-cni-cncf-io\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.705713 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.704984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-var-lib-cni-multus\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/102e4f9f-8d5c-40d6-8b00-7f1a13888508-system-cni-dir\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-system-cni-dir\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-run-netns\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-systemd\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-sys\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33f1203a-dd9a-46be-96bd-47184ae3fccf-host\") pod \"node-ca-n5hmv\" (UID: \"33f1203a-dd9a-46be-96bd-47184ae3fccf\") " pod="openshift-image-registry/node-ca-n5hmv"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705139    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/102e4f9f-8d5c-40d6-8b00-7f1a13888508-system-cni-dir\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705160    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-var-lib-kubelet\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705185    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-run-k8s-cni-cncf-io\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/102e4f9f-8d5c-40d6-8b00-7f1a13888508-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-run-netns\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-var-lib-cni-multus\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-kubelet\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.706376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705264    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33f1203a-dd9a-46be-96bd-47184ae3fccf-host\") pod \"node-ca-n5hmv\" (UID: \"33f1203a-dd9a-46be-96bd-47184ae3fccf\") " pod="openshift-image-registry/node-ca-n5hmv"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-system-cni-dir\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705305    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-kubelet\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-device-dir\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33f1203a-dd9a-46be-96bd-47184ae3fccf-serviceca\") pod \"node-ca-n5hmv\" (UID: \"33f1203a-dd9a-46be-96bd-47184ae3fccf\") " pod="openshift-image-registry/node-ca-n5hmv"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705326    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-var-lib-kubelet\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-systemd-units\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705366    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-sysctl-conf\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705369    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-slash\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-host\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705441    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-var-lib-cni-bin\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705509    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-cni-netd\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ab13120-6b0f-4805-bc85-bb1ee7384987-dbus\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705550    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-multus-cni-dir\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.707194 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705545    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-multus-conf-dir\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705590    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-sys\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705590    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-os-release\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705623    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-etc-openvswitch\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-kubernetes\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmx5z\" (UniqueName: \"kubernetes.io/projected/df44767e-e243-4004-a43e-4d2982ddd197-kube-api-access-pmx5z\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/102e4f9f-8d5c-40d6-8b00-7f1a13888508-cnibin\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.705947    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-cni-binary-copy\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m856h\" (UniqueName: \"kubernetes.io/projected/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-kube-api-access-m856h\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:02.706046    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-hostroot\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706100    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/102e4f9f-8d5c-40d6-8b00-7f1a13888508-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706105    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-ovnkube-script-lib\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706129    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-multus-daemon-config\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.707932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706179    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4skps\" (UniqueName: \"kubernetes.io/projected/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-kube-api-access-4skps\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706183    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/102e4f9f-8d5c-40d6-8b00-7f1a13888508-cnibin\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:02.706195    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret podName:3ab13120-6b0f-4805-bc85-bb1ee7384987 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:03.206175652 +0000 UTC m=+3.044274245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret") pod "global-pull-secret-syncer-rw9rr" (UID: "3ab13120-6b0f-4805-bc85-bb1ee7384987") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706225    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-node-log\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-log-socket\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thw9d\" (UniqueName: \"kubernetes.io/projected/c840916c-ce29-4f39-8f28-b2c462411b72-kube-api-access-thw9d\") pod \"iptables-alerter-2wmxz\" (UID: \"c840916c-ce29-4f39-8f28-b2c462411b72\") " pod="openshift-network-operator/iptables-alerter-2wmxz"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-node-log\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-kubernetes\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-hostroot\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706420    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-host\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706459    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/102e4f9f-8d5c-40d6-8b00-7f1a13888508-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a052b9a-0d1c-4b01-8b75-1f435ec83841-tmp-dir\") pod \"node-resolver-mn8bv\" (UID: \"6a052b9a-0d1c-4b01-8b75-1f435ec83841\") " pod="openshift-dns/node-resolver-mn8bv"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-sys-fs\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-sysctl-conf\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.708698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-log-socket\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-run-ovn\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706617    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-run-ovn\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706632    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-sys-fs\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706678    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-sysconfig\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/102e4f9f-8d5c-40d6-8b00-7f1a13888508-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706723    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-ovnkube-config\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-env-overrides\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-modprobe-d\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706869    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a052b9a-0d1c-4b01-8b75-1f435ec83841-tmp-dir\") pod \"node-resolver-mn8bv\" (UID: \"6a052b9a-0d1c-4b01-8b75-1f435ec83841\") " pod="openshift-dns/node-resolver-mn8bv"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-multus-daemon-config\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c840916c-ce29-4f39-8f28-b2c462411b72-host-slash\") pod \"iptables-alerter-2wmxz\" (UID: \"c840916c-ce29-4f39-8f28-b2c462411b72\") " pod="openshift-network-operator/iptables-alerter-2wmxz"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706967    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/013cc0d4-fe49-43cd-9fb5-1041e0b0001d-agent-certs\") pod \"konnectivity-agent-kj6v2\" (UID: \"013cc0d4-fe49-43cd-9fb5-1041e0b0001d\") " pod="kube-system/konnectivity-agent-kj6v2"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-etc-modprobe-d\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.706994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/013cc0d4-fe49-43cd-9fb5-1041e0b0001d-konnectivity-ca\") pod \"konnectivity-agent-kj6v2\" (UID: \"013cc0d4-fe49-43cd-9fb5-1041e0b0001d\") " pod="kube-system/konnectivity-agent-kj6v2"
Jul 20 09:57:02.709511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-run-multus-certs\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c840916c-ce29-4f39-8f28-b2c462411b72-host-slash\") pod \"iptables-alerter-2wmxz\" (UID: \"c840916c-ce29-4f39-8f28-b2c462411b72\") " pod="openshift-network-operator/iptables-alerter-2wmxz"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-var-lib-openvswitch\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-env-overrides\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707158    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-var-lib-kubelet\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d85828e-89be-41f7-a3e6-9011e4280223-tmp\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707260    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-ovnkube-config\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/102e4f9f-8d5c-40d6-8b00-7f1a13888508-os-release\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707308    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-var-lib-openvswitch\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/102e4f9f-8d5c-40d6-8b00-7f1a13888508-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707339    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a052b9a-0d1c-4b01-8b75-1f435ec83841-hosts-file\") pod \"node-resolver-mn8bv\" (UID: \"6a052b9a-0d1c-4b01-8b75-1f435ec83841\") " pod="openshift-dns/node-resolver-mn8bv"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707357    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-var-lib-kubelet\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707374    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s879n\" (UniqueName: \"kubernetes.io/projected/6a052b9a-0d1c-4b01-8b75-1f435ec83841-kube-api-access-s879n\") pod \"node-resolver-mn8bv\" (UID: \"6a052b9a-0d1c-4b01-8b75-1f435ec83841\") " pod="openshift-dns/node-resolver-mn8bv"
Jul 20 09:57:02.710275 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:02.707448    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:02.710962 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:02.707503    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs podName:df44767e-e243-4004-a43e-4d2982ddd197 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:03.207488553 +0000 UTC m=+3.045587122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs") pod "network-metrics-daemon-n4x2n" (UID: "df44767e-e243-4004-a43e-4d2982ddd197") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:02.710962 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-host-run-multus-certs\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.710962 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707552    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/013cc0d4-fe49-43cd-9fb5-1041e0b0001d-konnectivity-ca\") pod \"konnectivity-agent-kj6v2\" (UID: \"013cc0d4-fe49-43cd-9fb5-1041e0b0001d\") " pod="kube-system/konnectivity-agent-kj6v2"
Jul 20 09:57:02.710962 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/102e4f9f-8d5c-40d6-8b00-7f1a13888508-os-release\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.710962 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-run-netns\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.710962 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707630    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-run\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.710962 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707714    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d85828e-89be-41f7-a3e6-9011e4280223-run\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.710962 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707748    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-host-run-netns\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.710962 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.707938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a052b9a-0d1c-4b01-8b75-1f435ec83841-hosts-file\") pod \"node-resolver-mn8bv\" (UID: \"6a052b9a-0d1c-4b01-8b75-1f435ec83841\") " pod="openshift-dns/node-resolver-mn8bv"
Jul 20 09:57:02.710962 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.708366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/102e4f9f-8d5c-40d6-8b00-7f1a13888508-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.710962 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.709011    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-ovn-node-metrics-cert\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.714391 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.714342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d85828e-89be-41f7-a3e6-9011e4280223-tmp\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.714683 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.714646    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d85828e-89be-41f7-a3e6-9011e4280223-etc-tuned\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.714888 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.714866    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/013cc0d4-fe49-43cd-9fb5-1041e0b0001d-agent-certs\") pod \"konnectivity-agent-kj6v2\" (UID: \"013cc0d4-fe49-43cd-9fb5-1041e0b0001d\") " pod="kube-system/konnectivity-agent-kj6v2"
Jul 20 09:57:02.716684 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:02.716103    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:02.716684 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:02.716146    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:02.716684 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:02.716161    2561 projected.go:196] Error preparing data for projected volume kube-api-access-z8h5s for pod openshift-network-diagnostics/network-check-target-fcpp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:02.716684 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:02.716263    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s podName:ffd754c6-58d4-429e-aa73-9e3cd0f1626e nodeName:}" failed. No retries permitted until 2026-07-20 09:57:03.216247163 +0000 UTC m=+3.054345755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z8h5s" (UniqueName: "kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s") pod "network-check-target-fcpp5" (UID: "ffd754c6-58d4-429e-aa73-9e3cd0f1626e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:02.719509 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.719454    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c54h\" (UniqueName: \"kubernetes.io/projected/b1fd10ce-1d5a-4844-81d4-ef2e75cea222-kube-api-access-4c54h\") pod \"ovnkube-node-kxc68\" (UID: \"b1fd10ce-1d5a-4844-81d4-ef2e75cea222\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.720270 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.720248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbggz\" (UniqueName: \"kubernetes.io/projected/8d85828e-89be-41f7-a3e6-9011e4280223-kube-api-access-pbggz\") pod \"tuned-qsb9p\" (UID: \"8d85828e-89be-41f7-a3e6-9011e4280223\") " pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.720759 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.720670    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krnck\" (UniqueName: \"kubernetes.io/projected/33f1203a-dd9a-46be-96bd-47184ae3fccf-kube-api-access-krnck\") pod \"node-ca-n5hmv\" (UID: \"33f1203a-dd9a-46be-96bd-47184ae3fccf\") " pod="openshift-image-registry/node-ca-n5hmv"
Jul 20 09:57:02.720823 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.720764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4skps\" (UniqueName: \"kubernetes.io/projected/0787b82a-7c63-40a4-af72-91cfa5bfd7e4-kube-api-access-4skps\") pod \"multus-l8xwg\" (UID: \"0787b82a-7c63-40a4-af72-91cfa5bfd7e4\") " pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.721704 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.721681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmx5z\" (UniqueName: \"kubernetes.io/projected/df44767e-e243-4004-a43e-4d2982ddd197-kube-api-access-pmx5z\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:02.723442 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.722608    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcsdr\" (UniqueName: \"kubernetes.io/projected/102e4f9f-8d5c-40d6-8b00-7f1a13888508-kube-api-access-vcsdr\") pod \"multus-additional-cni-plugins-n8s76\" (UID: \"102e4f9f-8d5c-40d6-8b00-7f1a13888508\") " pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.723662 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.723627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m856h\" (UniqueName: \"kubernetes.io/projected/f4cf224b-7d2a-4c4e-a68d-a2aa80cda639-kube-api-access-m856h\") pod \"aws-ebs-csi-driver-node-92ktw\" (UID: \"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.727682 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.727354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s879n\" (UniqueName: \"kubernetes.io/projected/6a052b9a-0d1c-4b01-8b75-1f435ec83841-kube-api-access-s879n\") pod \"node-resolver-mn8bv\" (UID: \"6a052b9a-0d1c-4b01-8b75-1f435ec83841\") " pod="openshift-dns/node-resolver-mn8bv"
Jul 20 09:57:02.730372 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.729857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thw9d\" (UniqueName: \"kubernetes.io/projected/c840916c-ce29-4f39-8f28-b2c462411b72-kube-api-access-thw9d\") pod \"iptables-alerter-2wmxz\" (UID: \"c840916c-ce29-4f39-8f28-b2c462411b72\") " pod="openshift-network-operator/iptables-alerter-2wmxz"
Jul 20 09:57:02.887052 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.886983    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l8xwg"
Jul 20 09:57:02.899127 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.899091    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:02.911093 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.911072    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qsb9p"
Jul 20 09:57:02.919651 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.919630    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n8s76"
Jul 20 09:57:02.927331 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.927311    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n5hmv"
Jul 20 09:57:02.935010 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.934991    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mn8bv"
Jul 20 09:57:02.943630 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.943611    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw"
Jul 20 09:57:02.958236 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.958219    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2wmxz"
Jul 20 09:57:02.964834 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:02.964817    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kj6v2"
Jul 20 09:57:03.210693 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.210612    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:03.210693 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.210666    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:03.210889 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:03.210779    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:03.210889 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:03.210794    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:03.210889 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:03.210857    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret podName:3ab13120-6b0f-4805-bc85-bb1ee7384987 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:04.210835699 +0000 UTC m=+4.048934263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret") pod "global-pull-secret-syncer-rw9rr" (UID: "3ab13120-6b0f-4805-bc85-bb1ee7384987") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:03.210889 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:03.210879    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs podName:df44767e-e243-4004-a43e-4d2982ddd197 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:04.210868582 +0000 UTC m=+4.048967145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs") pod "network-metrics-daemon-n4x2n" (UID: "df44767e-e243-4004-a43e-4d2982ddd197") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:03.311820 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.311779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h5s\" (UniqueName: \"kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s\") pod \"network-check-target-fcpp5\" (UID: \"ffd754c6-58d4-429e-aa73-9e3cd0f1626e\") " pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:03.311990 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:03.311970    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:03.312049 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:03.312003    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:03.312049 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:03.312016    2561 projected.go:196] Error preparing data for projected volume kube-api-access-z8h5s for pod openshift-network-diagnostics/network-check-target-fcpp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:03.312171 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:03.312079    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s podName:ffd754c6-58d4-429e-aa73-9e3cd0f1626e nodeName:}" failed. No retries permitted until 2026-07-20 09:57:04.312059363 +0000 UTC m=+4.150157940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8h5s" (UniqueName: "kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s") pod "network-check-target-fcpp5" (UID: "ffd754c6-58d4-429e-aa73-9e3cd0f1626e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:03.595279 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:03.595254    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f1203a_dd9a_46be_96bd_47184ae3fccf.slice/crio-573985e873d5a9a701b1e994deefa627119beb3fa2abc330f4a7ce8482a090a7 WatchSource:0}: Error finding container 573985e873d5a9a701b1e994deefa627119beb3fa2abc330f4a7ce8482a090a7: Status 404 returned error can't find the container with id 573985e873d5a9a701b1e994deefa627119beb3fa2abc330f4a7ce8482a090a7
Jul 20 09:57:03.597842 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:03.597813    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a052b9a_0d1c_4b01_8b75_1f435ec83841.slice/crio-11bcd24887cf08ffe4301b8f04e82ca36d08bc49e91fcf81e5979df8e6cf03bd WatchSource:0}: Error finding container 11bcd24887cf08ffe4301b8f04e82ca36d08bc49e91fcf81e5979df8e6cf03bd: Status 404 returned error can't find the container with id 11bcd24887cf08ffe4301b8f04e82ca36d08bc49e91fcf81e5979df8e6cf03bd
Jul 20 09:57:03.600437 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:03.600349    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fd10ce_1d5a_4844_81d4_ef2e75cea222.slice/crio-b29812c467bd8bc9383cedfd017242a26a998d7eaf6da9b69b6a58ab448da689 WatchSource:0}: Error finding container b29812c467bd8bc9383cedfd017242a26a998d7eaf6da9b69b6a58ab448da689: Status 404 returned error can't find the container with id b29812c467bd8bc9383cedfd017242a26a998d7eaf6da9b69b6a58ab448da689
Jul 20 09:57:03.601932 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:03.601888    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102e4f9f_8d5c_40d6_8b00_7f1a13888508.slice/crio-606c9f37126970e3010d353fa7d8635fa6cd2c782a779f89727768249290fa45 WatchSource:0}: Error finding container 606c9f37126970e3010d353fa7d8635fa6cd2c782a779f89727768249290fa45: Status 404 returned error can't find the container with id 606c9f37126970e3010d353fa7d8635fa6cd2c782a779f89727768249290fa45
Jul 20 09:57:03.605649 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:03.605627    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d85828e_89be_41f7_a3e6_9011e4280223.slice/crio-572abf85c53566ee2efd89740bde0e674becda390b3332df62607cf57b098673 WatchSource:0}: Error finding container 572abf85c53566ee2efd89740bde0e674becda390b3332df62607cf57b098673: Status 404 returned error can't find the container with id 572abf85c53566ee2efd89740bde0e674becda390b3332df62607cf57b098673
Jul 20 09:57:03.606905 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:03.606881    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0787b82a_7c63_40a4_af72_91cfa5bfd7e4.slice/crio-09121a0a50e5dc0aef156507314b28b0793527cbe1d363c2e5264eeff2444c07 WatchSource:0}: Error finding container 09121a0a50e5dc0aef156507314b28b0793527cbe1d363c2e5264eeff2444c07: Status 404 returned error can't find the container with id 09121a0a50e5dc0aef156507314b28b0793527cbe1d363c2e5264eeff2444c07
Jul 20 09:57:03.607913 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:03.607849    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013cc0d4_fe49_43cd_9fb5_1041e0b0001d.slice/crio-cc76ea9f28c79c050a5306b357703139d3012e86cbc56ec3a7416ee0169e8191 WatchSource:0}: Error finding container cc76ea9f28c79c050a5306b357703139d3012e86cbc56ec3a7416ee0169e8191: Status 404 returned error can't find the container with id cc76ea9f28c79c050a5306b357703139d3012e86cbc56ec3a7416ee0169e8191
Jul 20 09:57:03.609077 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:03.609059    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cf224b_7d2a_4c4e_a68d_a2aa80cda639.slice/crio-0edce877ba35e85454a087e5df8e83b15807b0f2c5b891bba5f46dfed54051f4 WatchSource:0}: Error finding container 0edce877ba35e85454a087e5df8e83b15807b0f2c5b891bba5f46dfed54051f4: Status 404 returned error can't find the container with id 0edce877ba35e85454a087e5df8e83b15807b0f2c5b891bba5f46dfed54051f4
Jul 20 09:57:03.611129 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:03.610153    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc840916c_ce29_4f39_8f28_b2c462411b72.slice/crio-bf512cb0217550c16cfe15738f416aba596ecce1e655b85a522fafbdf1915c2f WatchSource:0}: Error finding container bf512cb0217550c16cfe15738f416aba596ecce1e655b85a522fafbdf1915c2f: Status 404 returned error can't find the container with id bf512cb0217550c16cfe15738f416aba596ecce1e655b85a522fafbdf1915c2f
Jul 20 09:57:03.644842 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.644799    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 09:52:02 +0000 UTC" deadline="2028-01-11 08:22:21.230934265 +0000 UTC"
Jul 20 09:57:03.644842 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.644831    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12958h25m17.586107819s"
Jul 20 09:57:03.760748 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.760619    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:03.761088 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:03.760850    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:03.768901 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.768875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qsb9p" event={"ID":"8d85828e-89be-41f7-a3e6-9011e4280223","Type":"ContainerStarted","Data":"572abf85c53566ee2efd89740bde0e674becda390b3332df62607cf57b098673"}
Jul 20 09:57:03.770660 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.770637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-80.ec2.internal" event={"ID":"d8acf0f9821033101209753c1ff5ad61","Type":"ContainerStarted","Data":"e529cf11e5dbca0bbd578a191fad220435d485906012bf8efb1f27b88eea0dd7"}
Jul 20 09:57:03.772637 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.772616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2wmxz" event={"ID":"c840916c-ce29-4f39-8f28-b2c462411b72","Type":"ContainerStarted","Data":"bf512cb0217550c16cfe15738f416aba596ecce1e655b85a522fafbdf1915c2f"}
Jul 20 09:57:03.773784 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.773761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68" event={"ID":"b1fd10ce-1d5a-4844-81d4-ef2e75cea222","Type":"ContainerStarted","Data":"b29812c467bd8bc9383cedfd017242a26a998d7eaf6da9b69b6a58ab448da689"}
Jul 20 09:57:03.774821 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.774800    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mn8bv" event={"ID":"6a052b9a-0d1c-4b01-8b75-1f435ec83841","Type":"ContainerStarted","Data":"11bcd24887cf08ffe4301b8f04e82ca36d08bc49e91fcf81e5979df8e6cf03bd"}
Jul 20 09:57:03.778425 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.778398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n5hmv" event={"ID":"33f1203a-dd9a-46be-96bd-47184ae3fccf","Type":"ContainerStarted","Data":"573985e873d5a9a701b1e994deefa627119beb3fa2abc330f4a7ce8482a090a7"}
Jul 20 09:57:03.780376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.780351    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw" event={"ID":"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639","Type":"ContainerStarted","Data":"0edce877ba35e85454a087e5df8e83b15807b0f2c5b891bba5f46dfed54051f4"}
Jul 20 09:57:03.783503 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.783479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kj6v2" event={"ID":"013cc0d4-fe49-43cd-9fb5-1041e0b0001d","Type":"ContainerStarted","Data":"cc76ea9f28c79c050a5306b357703139d3012e86cbc56ec3a7416ee0169e8191"}
Jul 20 09:57:03.784495 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.784471    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8s76" event={"ID":"102e4f9f-8d5c-40d6-8b00-7f1a13888508","Type":"ContainerStarted","Data":"606c9f37126970e3010d353fa7d8635fa6cd2c782a779f89727768249290fa45"}
Jul 20 09:57:03.785547 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.785530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l8xwg" event={"ID":"0787b82a-7c63-40a4-af72-91cfa5bfd7e4","Type":"ContainerStarted","Data":"09121a0a50e5dc0aef156507314b28b0793527cbe1d363c2e5264eeff2444c07"}
Jul 20 09:57:03.786860 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:03.786807    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-80.ec2.internal" podStartSLOduration=1.786790392 podStartE2EDuration="1.786790392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:03.786683407 +0000 UTC m=+3.624781992" watchObservedRunningTime="2026-07-20 09:57:03.786790392 +0000 UTC m=+3.624888979"
Jul 20 09:57:04.218255 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:04.218220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:04.218433 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:04.218304    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:04.218433 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:04.218385    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:04.218433 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:04.218418    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:04.218609 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:04.218460    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs podName:df44767e-e243-4004-a43e-4d2982ddd197 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:06.218436099 +0000 UTC m=+6.056534662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs") pod "network-metrics-daemon-n4x2n" (UID: "df44767e-e243-4004-a43e-4d2982ddd197") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:04.218609 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:04.218482    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret podName:3ab13120-6b0f-4805-bc85-bb1ee7384987 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:06.218472253 +0000 UTC m=+6.056570822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret") pod "global-pull-secret-syncer-rw9rr" (UID: "3ab13120-6b0f-4805-bc85-bb1ee7384987") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:04.319249 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:04.319214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h5s\" (UniqueName: \"kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s\") pod \"network-check-target-fcpp5\" (UID: \"ffd754c6-58d4-429e-aa73-9e3cd0f1626e\") " pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:04.319440 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:04.319390    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:04.319440 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:04.319408    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:04.319440 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:04.319420    2561 projected.go:196] Error preparing data for projected volume kube-api-access-z8h5s for pod openshift-network-diagnostics/network-check-target-fcpp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:04.319594 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:04.319475    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s podName:ffd754c6-58d4-429e-aa73-9e3cd0f1626e nodeName:}" failed. No retries permitted until 2026-07-20 09:57:06.319456436 +0000 UTC m=+6.157555020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8h5s" (UniqueName: "kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s") pod "network-check-target-fcpp5" (UID: "ffd754c6-58d4-429e-aa73-9e3cd0f1626e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:04.644999 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:04.644936    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 09:52:02 +0000 UTC" deadline="2027-12-26 06:06:24.635262773 +0000 UTC"
Jul 20 09:57:04.644999 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:04.644995    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12572h9m19.990272399s"
Jul 20 09:57:04.762882 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:04.762849    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:04.763373 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:04.762987    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:04.763373 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:04.763086    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:04.763373 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:04.763220    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:04.806181 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:04.805047    2561 generic.go:356] "Generic (PLEG): container finished" podID="00c0661b8872f25c32c9ae0016e0b8ef" containerID="13eb84df96033e6a9b8c9b5df1a372fa66aa29c8b71e21469bd969713dcea2c5" exitCode=0
Jul 20 09:57:04.806181 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:04.805954    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal" event={"ID":"00c0661b8872f25c32c9ae0016e0b8ef","Type":"ContainerDied","Data":"13eb84df96033e6a9b8c9b5df1a372fa66aa29c8b71e21469bd969713dcea2c5"}
Jul 20 09:57:05.760406 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:05.760318    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:05.760564 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:05.760444    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:05.837199 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:05.837166    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal" event={"ID":"00c0661b8872f25c32c9ae0016e0b8ef","Type":"ContainerStarted","Data":"88ad1bdfd3cafec15c4afc1ddefbbd82474f206658d3c9886e1abd31f61aa76b"}
Jul 20 09:57:06.234929 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:06.234894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:06.235184 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:06.234980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:06.235184 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:06.235130    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:06.235184 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:06.235186    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret podName:3ab13120-6b0f-4805-bc85-bb1ee7384987 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:10.235168275 +0000 UTC m=+10.073266844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret") pod "global-pull-secret-syncer-rw9rr" (UID: "3ab13120-6b0f-4805-bc85-bb1ee7384987") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:06.235381 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:06.235251    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:06.235381 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:06.235284    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs podName:df44767e-e243-4004-a43e-4d2982ddd197 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:10.235272747 +0000 UTC m=+10.073371311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs") pod "network-metrics-daemon-n4x2n" (UID: "df44767e-e243-4004-a43e-4d2982ddd197") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:06.335487 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:06.335454    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h5s\" (UniqueName: \"kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s\") pod \"network-check-target-fcpp5\" (UID: \"ffd754c6-58d4-429e-aa73-9e3cd0f1626e\") " pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:06.335657 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:06.335623    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:06.335657 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:06.335643    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:06.335657 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:06.335655    2561 projected.go:196] Error preparing data for projected volume kube-api-access-z8h5s for pod openshift-network-diagnostics/network-check-target-fcpp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:06.335820 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:06.335713    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s podName:ffd754c6-58d4-429e-aa73-9e3cd0f1626e nodeName:}" failed. No retries permitted until 2026-07-20 09:57:10.335694547 +0000 UTC m=+10.173793113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8h5s" (UniqueName: "kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s") pod "network-check-target-fcpp5" (UID: "ffd754c6-58d4-429e-aa73-9e3cd0f1626e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:06.760712 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:06.760681    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:06.760871 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:06.760681    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:06.760871 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:06.760812    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:06.760984 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:06.760882    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:07.761041 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:07.761010    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:07.761512 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:07.761159    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:08.760417 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:08.760333    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:08.760417 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:08.760370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:08.760625 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:08.760468    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:08.760854 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:08.760819    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:09.760946 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:09.760853    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:09.761407 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:09.761012    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:10.268635 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:10.268596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:10.268826 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:10.268663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:10.268826 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:10.268778    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:10.268826 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:10.268795    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:10.268995 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:10.268866    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret podName:3ab13120-6b0f-4805-bc85-bb1ee7384987 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:18.268843184 +0000 UTC m=+18.106941762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret") pod "global-pull-secret-syncer-rw9rr" (UID: "3ab13120-6b0f-4805-bc85-bb1ee7384987") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:10.268995 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:10.268885    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs podName:df44767e-e243-4004-a43e-4d2982ddd197 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:18.268875449 +0000 UTC m=+18.106974016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs") pod "network-metrics-daemon-n4x2n" (UID: "df44767e-e243-4004-a43e-4d2982ddd197") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:10.370070 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:10.369962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h5s\" (UniqueName: \"kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s\") pod \"network-check-target-fcpp5\" (UID: \"ffd754c6-58d4-429e-aa73-9e3cd0f1626e\") " pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:10.370266 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:10.370236    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:10.370266 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:10.370257    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:10.370374 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:10.370270    2561 projected.go:196] Error preparing data for projected volume kube-api-access-z8h5s for pod openshift-network-diagnostics/network-check-target-fcpp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:10.370374 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:10.370333    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s podName:ffd754c6-58d4-429e-aa73-9e3cd0f1626e nodeName:}" failed. No retries permitted until 2026-07-20 09:57:18.370313869 +0000 UTC m=+18.208412456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8h5s" (UniqueName: "kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s") pod "network-check-target-fcpp5" (UID: "ffd754c6-58d4-429e-aa73-9e3cd0f1626e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:10.762840 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:10.762435    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:10.762840 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:10.762468    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:10.762840 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:10.762542    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:10.762840 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:10.762595    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:11.760735 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:11.760652    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:11.760911 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:11.760801    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:12.760891 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:12.760834    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:12.761332 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:12.760961    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:12.761332 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:12.761010    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:12.761332 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:12.761108    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:13.761095 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:13.761067    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:13.761477 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:13.761180    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:14.760440 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:14.760342    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:14.760672 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:14.760489    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:14.760672 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:14.760531    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:14.760672 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:14.760633    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:15.761210 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:15.761176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:15.761634 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:15.761309    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:16.760857 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:16.760824    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:16.761028 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:16.760950    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:16.761028 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:16.761007    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:16.761171 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:16.761138    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:17.760416 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:17.760386    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:17.760876 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:17.760500    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:18.329454 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:18.329427    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:18.329639 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:18.329487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:18.329639 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:18.329560    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:18.329639 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:18.329568    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:18.329639 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:18.329613    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs podName:df44767e-e243-4004-a43e-4d2982ddd197 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.329595207 +0000 UTC m=+34.167693788 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs") pod "network-metrics-daemon-n4x2n" (UID: "df44767e-e243-4004-a43e-4d2982ddd197") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:18.329639 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:18.329627    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret podName:3ab13120-6b0f-4805-bc85-bb1ee7384987 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.329620908 +0000 UTC m=+34.167719470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret") pod "global-pull-secret-syncer-rw9rr" (UID: "3ab13120-6b0f-4805-bc85-bb1ee7384987") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:18.430581 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:18.430553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h5s\" (UniqueName: \"kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s\") pod \"network-check-target-fcpp5\" (UID: \"ffd754c6-58d4-429e-aa73-9e3cd0f1626e\") " pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:18.430704 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:18.430680    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:18.430704 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:18.430701    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:18.430773 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:18.430713    2561 projected.go:196] Error preparing data for projected volume kube-api-access-z8h5s for pod openshift-network-diagnostics/network-check-target-fcpp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:18.430773 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:18.430757    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s podName:ffd754c6-58d4-429e-aa73-9e3cd0f1626e nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.430744908 +0000 UTC m=+34.268843471 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8h5s" (UniqueName: "kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s") pod "network-check-target-fcpp5" (UID: "ffd754c6-58d4-429e-aa73-9e3cd0f1626e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:18.761128 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:18.761086    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:18.761590 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:18.761227    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:18.761590 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:18.761302    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:18.761590 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:18.761413    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:19.760837 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:19.760805    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:19.761013 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:19.760922    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:20.762020 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:20.761987    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:20.762430 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:20.762070    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:20.762430 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:20.762096    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:20.762430 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:20.762188    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:21.761236 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:21.761201    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:21.761443 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:21.761344    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:22.760455 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.760293    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:22.760868 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.760293    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:22.760868 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:22.760527    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:22.760868 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:22.760632    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:22.868222 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.868041    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 09:57:22.868589 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.868566    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1fd10ce-1d5a-4844-81d4-ef2e75cea222" containerID="9a9cd04ddd92deda86954720f29116ecbc900e256849c3b120c25390330f2aa5" exitCode=1
Jul 20 09:57:22.868694 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.868629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68" event={"ID":"b1fd10ce-1d5a-4844-81d4-ef2e75cea222","Type":"ContainerStarted","Data":"31ab7bb21b47356218dbe504557e1e305230a9530e5068286fccac7dba2b782f"}
Jul 20 09:57:22.868754 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.868712    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68" event={"ID":"b1fd10ce-1d5a-4844-81d4-ef2e75cea222","Type":"ContainerStarted","Data":"a3c5c4b92a4e437bb49b84e021e5228590dc87c831f768ff2345931979daa564"}
Jul 20 09:57:22.868811 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.868761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68" event={"ID":"b1fd10ce-1d5a-4844-81d4-ef2e75cea222","Type":"ContainerStarted","Data":"2d130ef5354c25002d32e8fa369eadc26b58308eed9183a096ddbc0556f2530d"}
Jul 20 09:57:22.868811 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.868806    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68" event={"ID":"b1fd10ce-1d5a-4844-81d4-ef2e75cea222","Type":"ContainerDied","Data":"9a9cd04ddd92deda86954720f29116ecbc900e256849c3b120c25390330f2aa5"}
Jul 20 09:57:22.868899 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.868860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68" event={"ID":"b1fd10ce-1d5a-4844-81d4-ef2e75cea222","Type":"ContainerStarted","Data":"43ae8a261b6351e106f2126e410a4a289608a7e7941ddfe2bbdde2e9d7b38c27"}
Jul 20 09:57:22.870162 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.870138    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mn8bv" event={"ID":"6a052b9a-0d1c-4b01-8b75-1f435ec83841","Type":"ContainerStarted","Data":"c84de21a2fbe24dd72d60ad1d4deae5ecd5346c600377cdd2ff00790d437d910"}
Jul 20 09:57:22.871757 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.871729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n5hmv" event={"ID":"33f1203a-dd9a-46be-96bd-47184ae3fccf","Type":"ContainerStarted","Data":"5a89bedf2bc6ca15412888290c93d769e2c264462dc35ea182a9b2e081a4a3ae"}
Jul 20 09:57:22.873319 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.873299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw" event={"ID":"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639","Type":"ContainerStarted","Data":"6a2acbdbe26ffc59355c1674aedc1aa01ac47b4cfe5dfe79c4cf069baf986ad7"}
Jul 20 09:57:22.874869 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.874848    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kj6v2" event={"ID":"013cc0d4-fe49-43cd-9fb5-1041e0b0001d","Type":"ContainerStarted","Data":"a8b8176db7fdd9d1c9c7d85414a87c9291c55131d53e99f3d2c0894bc5a205db"}
Jul 20 09:57:22.878710 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.878684    2561 generic.go:356] "Generic (PLEG): container finished" podID="102e4f9f-8d5c-40d6-8b00-7f1a13888508" containerID="353cc7760e37fff69fa013f7b1d58cb25d751d15428e101c11794cb723fa3855" exitCode=0
Jul 20 09:57:22.878802 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.878745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8s76" event={"ID":"102e4f9f-8d5c-40d6-8b00-7f1a13888508","Type":"ContainerDied","Data":"353cc7760e37fff69fa013f7b1d58cb25d751d15428e101c11794cb723fa3855"}
Jul 20 09:57:22.881184 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.881163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qsb9p" event={"ID":"8d85828e-89be-41f7-a3e6-9011e4280223","Type":"ContainerStarted","Data":"6b616e84cc1a67091805184196bf331abe49774cda242227823e3e17ba3c5cd1"}
Jul 20 09:57:22.893475 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.893054    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-80.ec2.internal" podStartSLOduration=20.893017537 podStartE2EDuration="20.893017537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:05.853952893 +0000 UTC m=+5.692051479" watchObservedRunningTime="2026-07-20 09:57:22.893017537 +0000 UTC m=+22.731116119"
Jul 20 09:57:22.893924 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.893892    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mn8bv" podStartSLOduration=3.342702319 podStartE2EDuration="21.893880682s" totalImagesPullingTime="18.551178363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:22.893163791 +0000 UTC m=+22.731262402" watchObservedRunningTime="2026-07-20 09:57:22.893880682 +0000 UTC m=+22.731979267"
Jul 20 09:57:22.914540 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.914479    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qsb9p" podStartSLOduration=4.370279521 podStartE2EDuration="22.914464936s" totalImagesPullingTime="18.544185415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:22.914049208 +0000 UTC m=+22.752147794" watchObservedRunningTime="2026-07-20 09:57:22.914464936 +0000 UTC m=+22.752563520"
Jul 20 09:57:22.970353 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.970288    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n5hmv" podStartSLOduration=3.434706808 podStartE2EDuration="21.970267355s" totalImagesPullingTime="18.535560547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:22.934180595 +0000 UTC m=+22.772279181" watchObservedRunningTime="2026-07-20 09:57:22.970267355 +0000 UTC m=+22.808366011"
Jul 20 09:57:22.987639 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:22.987596    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kj6v2" podStartSLOduration=3.501298825 podStartE2EDuration="21.987581368s" totalImagesPullingTime="18.486282543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:22.986957465 +0000 UTC m=+22.825056052" watchObservedRunningTime="2026-07-20 09:57:22.987581368 +0000 UTC m=+22.825679992"
Jul 20 09:57:23.760542 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:23.760516    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:23.761138 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:23.760657    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:23.803755 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:23.803729    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 20 09:57:23.885864 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:23.885835    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2wmxz" event={"ID":"c840916c-ce29-4f39-8f28-b2c462411b72","Type":"ContainerStarted","Data":"8152f814034bbacf50979a1565e669cbf476df67b6203b7d44f39a819a5836d3"}
Jul 20 09:57:23.889481 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:23.889462    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 09:57:23.889903 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:23.889876    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68" event={"ID":"b1fd10ce-1d5a-4844-81d4-ef2e75cea222","Type":"ContainerStarted","Data":"3aca895f3f47ac4ab7720563c011c4c1eb7ebdd4445738bc4c6d825860247a54"}
Jul 20 09:57:23.891878 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:23.891856    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw" event={"ID":"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639","Type":"ContainerStarted","Data":"8fe90dc3535153dd51517d8254ddf451d9288c8ed8378acda978a4089732123b"}
Jul 20 09:57:23.893514 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:23.893456    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l8xwg" event={"ID":"0787b82a-7c63-40a4-af72-91cfa5bfd7e4","Type":"ContainerStarted","Data":"b3d285ccd7a850e651f184c26c33a64548eec42e345108b68af33792d38cae7c"}
Jul 20 09:57:23.903294 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:23.903253    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2wmxz" podStartSLOduration=4.358940908 podStartE2EDuration="22.903239981s" totalImagesPullingTime="18.544299073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:23.90306301 +0000 UTC m=+23.741161595" watchObservedRunningTime="2026-07-20 09:57:23.903239981 +0000 UTC m=+23.741338588"
Jul 20 09:57:23.921553 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:23.921518    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l8xwg" podStartSLOduration=4.698825361 podStartE2EDuration="23.921508882s" totalImagesPullingTime="19.222683521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:23.921326018 +0000 UTC m=+23.759424600" watchObservedRunningTime="2026-07-20 09:57:23.921508882 +0000 UTC m=+23.759607467"
Jul 20 09:57:24.730409 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:24.730304    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T09:57:23.803750492Z","UUID":"2692e341-001a-4d89-afa8-5ace13d43da2","Handler":null,"Name":"","Endpoint":""}
Jul 20 09:57:24.731915 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:24.731895    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 20 09:57:24.732044 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:24.731933    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 20 09:57:24.760986 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:24.760962    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:24.761401 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:24.760971    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:24.761401 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:24.761089    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:24.761401 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:24.761223    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:25.281159 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:25.280888    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kj6v2"
Jul 20 09:57:25.281607 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:25.281586    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kj6v2"
Jul 20 09:57:25.479766 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:25.479735    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kj6v2"
Jul 20 09:57:25.480269 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:25.480251    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kj6v2"
Jul 20 09:57:25.761273 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:25.761244    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:25.761676 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:25.761377    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:25.900056 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:25.900007    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw" event={"ID":"f4cf224b-7d2a-4c4e-a68d-a2aa80cda639","Type":"ContainerStarted","Data":"ea08f0bfe9abd2b4e9b3efde60b746edf773bdb427cd62a4715d998d08eab34c"}
Jul 20 09:57:25.903394 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:25.903373    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 09:57:25.903847 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:25.903823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68" event={"ID":"b1fd10ce-1d5a-4844-81d4-ef2e75cea222","Type":"ContainerStarted","Data":"e577fa6481f57e9f503514d283cf397e02b1fbca461d07b7bc255eda87c12416"}
Jul 20 09:57:25.919781 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:25.919744    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92ktw" podStartSLOduration=4.140192857 podStartE2EDuration="24.919730291s" totalImagesPullingTime="20.779537434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:25.918975168 +0000 UTC m=+25.757073753" watchObservedRunningTime="2026-07-20 09:57:25.919730291 +0000 UTC m=+25.757828875"
Jul 20 09:57:26.761277 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:26.761241    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:26.761871 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:26.761380    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:26.761871 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:26.761416    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:26.761871 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:26.761503    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:27.760432 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:27.760403    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:27.760564 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:27.760540    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:27.911221 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:27.911196    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 09:57:27.911865 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:27.911583    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68" event={"ID":"b1fd10ce-1d5a-4844-81d4-ef2e75cea222","Type":"ContainerStarted","Data":"0e70cb07447db6c15bccab19e251dd5e05fc8e3dacf3441540c3b62b7e9753f2"}
Jul 20 09:57:27.911933 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:27.911873    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:27.912002 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:27.911982    2561 scope.go:117] "RemoveContainer" containerID="9a9cd04ddd92deda86954720f29116ecbc900e256849c3b120c25390330f2aa5"
Jul 20 09:57:27.913498 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:27.913473    2561 generic.go:356] "Generic (PLEG): container finished" podID="102e4f9f-8d5c-40d6-8b00-7f1a13888508" containerID="6bc8dc3d7054bac0d09e243948a65c4ee03445374401496c9024847f4291014e" exitCode=0
Jul 20 09:57:27.913599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:27.913513    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8s76" event={"ID":"102e4f9f-8d5c-40d6-8b00-7f1a13888508","Type":"ContainerDied","Data":"6bc8dc3d7054bac0d09e243948a65c4ee03445374401496c9024847f4291014e"}
Jul 20 09:57:27.926920 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:27.926902    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:28.761102 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:28.761066    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:28.761276 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:28.761183    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:28.761276 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:28.761217    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:28.761383 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:28.761292    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:28.920040 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:28.920013    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 09:57:28.920515 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:28.920426    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68" event={"ID":"b1fd10ce-1d5a-4844-81d4-ef2e75cea222","Type":"ContainerStarted","Data":"6ef1211aa5ba00e35e285d528a28cd227130534a6f25f3eff41cc6fff62f7181"}
Jul 20 09:57:28.920668 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:28.920648    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:28.920747 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:28.920723    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:28.938362 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:28.938338    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:57:28.950565 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:28.950518    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68" podStartSLOduration=10.235353548 podStartE2EDuration="28.950501221s" totalImagesPullingTime="18.715147673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:28.948648796 +0000 UTC m=+28.786747407" watchObservedRunningTime="2026-07-20 09:57:28.950501221 +0000 UTC m=+28.788599807"
Jul 20 09:57:29.232778 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:29.232748    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rw9rr"]
Jul 20 09:57:29.232961 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:29.232838    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:29.232961 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:29.232920    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:29.235210 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:29.235188    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fcpp5"]
Jul 20 09:57:29.235314 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:29.235260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:29.235355 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:29.235330    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:29.238170 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:29.238149    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n4x2n"]
Jul 20 09:57:29.238249 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:29.238239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:29.238331 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:29.238315    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:29.924185 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:29.924153    2561 generic.go:356] "Generic (PLEG): container finished" podID="102e4f9f-8d5c-40d6-8b00-7f1a13888508" containerID="1c250c03044bf679fbd0721ae2488692721781a4baca1d47b1056b950dfd1f4c" exitCode=0
Jul 20 09:57:29.924596 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:29.924249    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8s76" event={"ID":"102e4f9f-8d5c-40d6-8b00-7f1a13888508","Type":"ContainerDied","Data":"1c250c03044bf679fbd0721ae2488692721781a4baca1d47b1056b950dfd1f4c"}
Jul 20 09:57:30.760989 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:30.760957    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:30.761183 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:30.761055    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:30.761183 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:30.761060    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:30.761307 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:30.761200    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:30.761307 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:30.761214    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:30.761485 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:30.761339    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:31.933200 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:31.933167    2561 generic.go:356] "Generic (PLEG): container finished" podID="102e4f9f-8d5c-40d6-8b00-7f1a13888508" containerID="7448483376fab9d184f83672b13026cadff120c74a2559ecb22db0c54917372a" exitCode=0
Jul 20 09:57:31.933657 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:31.933242    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8s76" event={"ID":"102e4f9f-8d5c-40d6-8b00-7f1a13888508","Type":"ContainerDied","Data":"7448483376fab9d184f83672b13026cadff120c74a2559ecb22db0c54917372a"}
Jul 20 09:57:32.764389 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:32.764365    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:32.764536 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:32.764472    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:32.764620 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:32.764564    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:32.764688 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:32.764668    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:32.764742 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:32.764698    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:32.764814 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:32.764797    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:34.350753 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:34.350711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:34.351329 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:34.350777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:34.351329 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:34.350876    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:34.351329 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:34.350881    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:34.351329 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:34.350939    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs podName:df44767e-e243-4004-a43e-4d2982ddd197 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:06.350922853 +0000 UTC m=+66.189021416 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs") pod "network-metrics-daemon-n4x2n" (UID: "df44767e-e243-4004-a43e-4d2982ddd197") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:34.351329 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:34.350954    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret podName:3ab13120-6b0f-4805-bc85-bb1ee7384987 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:06.350947708 +0000 UTC m=+66.189046271 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret") pod "global-pull-secret-syncer-rw9rr" (UID: "3ab13120-6b0f-4805-bc85-bb1ee7384987") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:34.451294 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:34.451264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h5s\" (UniqueName: \"kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s\") pod \"network-check-target-fcpp5\" (UID: \"ffd754c6-58d4-429e-aa73-9e3cd0f1626e\") " pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:34.451446 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:34.451425    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:34.451446 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:34.451446    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:34.451519 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:34.451459    2561 projected.go:196] Error preparing data for projected volume kube-api-access-z8h5s for pod openshift-network-diagnostics/network-check-target-fcpp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:34.451555 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:34.451519    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s podName:ffd754c6-58d4-429e-aa73-9e3cd0f1626e nodeName:}" failed. No retries permitted until 2026-07-20 09:58:06.45150232 +0000 UTC m=+66.289600889 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8h5s" (UniqueName: "kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s") pod "network-check-target-fcpp5" (UID: "ffd754c6-58d4-429e-aa73-9e3cd0f1626e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:34.760783 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:34.760748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:34.760968 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:34.760748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:34.760968 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:34.760896    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4x2n" podUID="df44767e-e243-4004-a43e-4d2982ddd197"
Jul 20 09:57:34.761103 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:34.760992    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fcpp5" podUID="ffd754c6-58d4-429e-aa73-9e3cd0f1626e"
Jul 20 09:57:34.761103 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:34.760748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:34.761219 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:34.761103    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw9rr" podUID="3ab13120-6b0f-4805-bc85-bb1ee7384987"
Jul 20 09:57:35.478427 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.478395    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-80.ec2.internal" event="NodeReady"
Jul 20 09:57:35.478984 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.478535    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 20 09:57:35.525081 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.525010    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm"]
Jul 20 09:57:35.527670 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.527650    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm"
Jul 20 09:57:35.535208 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.535184    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 20 09:57:35.535504 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.535244    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jzlgb\""
Jul 20 09:57:35.535599 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.535585    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 20 09:57:35.535673 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.535618    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 20 09:57:35.535787 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.535769    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 20 09:57:35.537503 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.537382    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"]
Jul 20 09:57:35.539468 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.539450    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"]
Jul 20 09:57:35.539582 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.539563    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.541387 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.541370    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c9f447655-c7crr"]
Jul 20 09:57:35.541496 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.541479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:35.543974 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.543958    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.544500 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.544482    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 20 09:57:35.544597 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.544487    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 20 09:57:35.544853 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.544800    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 20 09:57:35.545386 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.545368    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 20 09:57:35.545480 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.545430    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 20 09:57:35.546161 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.546143    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m6fz5\""
Jul 20 09:57:35.552792 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.552768    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 20 09:57:35.552931 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.552903    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 20 09:57:35.553073 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.553058    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 20 09:57:35.559670 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.559650    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 20 09:57:35.560449 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.560433    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bzw96"]
Jul 20 09:57:35.562439 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.562421    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lx7wv"]
Jul 20 09:57:35.562569 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.562552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bzw96"
Jul 20 09:57:35.564654 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.564638    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:35.567175 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:35.567153    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-mjpn5\" is forbidden: User \"system:node:ip-10-0-134-80.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-134-80.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mjpn5\"" type="*v1.Secret"
Jul 20 09:57:35.567275 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:35.567208    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-134-80.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-134-80.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 20 09:57:35.567275 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:35.567233    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-134-80.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-134-80.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 20 09:57:35.567705 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:35.567686    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"dns-default\" is forbidden: User \"system:node:ip-10-0-134-80.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-134-80.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap"
Jul 20 09:57:35.567937 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.567920    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 20 09:57:35.569141 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.569104    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7r5mf\""
Jul 20 09:57:35.569433 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.569189    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 20 09:57:35.569838 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.569821    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"]
Jul 20 09:57:35.570764 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.570733    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"]
Jul 20 09:57:35.586849 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.586827    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bzw96"]
Jul 20 09:57:35.589897 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.589810    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lx7wv"]
Jul 20 09:57:35.596771 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.596746    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c9f447655-c7crr"]
Jul 20 09:57:35.608624 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.608604    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm"]
Jul 20 09:57:35.659595 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0314ce0d-5518-438f-858a-77a424b8dd4c-ca\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.659595 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp9m5\" (UniqueName: \"kubernetes.io/projected/0314ce0d-5518-438f-858a-77a424b8dd4c-kube-api-access-rp9m5\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.659813 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71be3c38-364e-4ace-8643-1324c5176e0d-config-volume\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:35.659813 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659631    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:35.659813 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659657    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c5b7fd4-f72c-4e24-a6c0-19b1182875ce-tmp\") pod \"klusterlet-addon-workmgr-5674f48f96-47stz\" (UID: \"9c5b7fd4-f72c-4e24-a6c0-19b1182875ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:35.659813 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659701    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf17a32f-809e-4273-ac73-17a8c358bb29-installation-pull-secrets\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.659813 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsz5l\" (UniqueName: \"kubernetes.io/projected/dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5-kube-api-access-jsz5l\") pod \"ingress-canary-bzw96\" (UID: \"dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5\") " pod="openshift-ingress-canary/ingress-canary-bzw96"
Jul 20 09:57:35.659813 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0314ce0d-5518-438f-858a-77a424b8dd4c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.660142 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659837    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0314ce0d-5518-438f-858a-77a424b8dd4c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.660142 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf17a32f-809e-4273-ac73-17a8c358bb29-image-registry-private-configuration\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.660142 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9x47\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-kube-api-access-k9x47\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.660142 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0314ce0d-5518-438f-858a-77a424b8dd4c-hub\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.660142 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.660142 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0314ce0d-5518-438f-858a-77a424b8dd4c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.660142 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9c5b7fd4-f72c-4e24-a6c0-19b1182875ce-klusterlet-config\") pod \"klusterlet-addon-workmgr-5674f48f96-47stz\" (UID: \"9c5b7fd4-f72c-4e24-a6c0-19b1182875ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:35.660142 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.659981    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf17a32f-809e-4273-ac73-17a8c358bb29-trusted-ca\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.660142 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.660007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqqnj\" (UniqueName: \"kubernetes.io/projected/71be3c38-364e-4ace-8643-1324c5176e0d-kube-api-access-xqqnj\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:35.660142 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.660074    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf17a32f-809e-4273-ac73-17a8c358bb29-ca-trust-extracted\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.660142 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.660092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5-cert\") pod \"ingress-canary-bzw96\" (UID: \"dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5\") " pod="openshift-ingress-canary/ingress-canary-bzw96"
Jul 20 09:57:35.660142 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.660127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s7rl\" (UniqueName: \"kubernetes.io/projected/8c786270-e174-406a-a50c-9672bad97bf2-kube-api-access-4s7rl\") pod \"managed-serviceaccount-addon-agent-79c996d7f8-97xdm\" (UID: \"8c786270-e174-406a-a50c-9672bad97bf2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm"
Jul 20 09:57:35.660653 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.660169    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4tph\" (UniqueName: \"kubernetes.io/projected/9c5b7fd4-f72c-4e24-a6c0-19b1182875ce-kube-api-access-k4tph\") pod \"klusterlet-addon-workmgr-5674f48f96-47stz\" (UID: \"9c5b7fd4-f72c-4e24-a6c0-19b1182875ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:35.660653 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.660229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c786270-e174-406a-a50c-9672bad97bf2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79c996d7f8-97xdm\" (UID: \"8c786270-e174-406a-a50c-9672bad97bf2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm"
Jul 20 09:57:35.660653 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.660258    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-certificates\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.660653 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.660281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-bound-sa-token\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.660653 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.660304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71be3c38-364e-4ace-8643-1324c5176e0d-tmp-dir\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:35.761220 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761181    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0314ce0d-5518-438f-858a-77a424b8dd4c-ca\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.761373 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp9m5\" (UniqueName: \"kubernetes.io/projected/0314ce0d-5518-438f-858a-77a424b8dd4c-kube-api-access-rp9m5\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.761373 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71be3c38-364e-4ace-8643-1324c5176e0d-config-volume\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:35.761373 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:35.761373 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c5b7fd4-f72c-4e24-a6c0-19b1182875ce-tmp\") pod \"klusterlet-addon-workmgr-5674f48f96-47stz\" (UID: \"9c5b7fd4-f72c-4e24-a6c0-19b1182875ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:35.761373 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf17a32f-809e-4273-ac73-17a8c358bb29-installation-pull-secrets\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.761373 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsz5l\" (UniqueName: \"kubernetes.io/projected/dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5-kube-api-access-jsz5l\") pod \"ingress-canary-bzw96\" (UID: \"dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5\") " pod="openshift-ingress-canary/ingress-canary-bzw96"
Jul 20 09:57:35.761648 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0314ce0d-5518-438f-858a-77a424b8dd4c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.761648 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761423    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0314ce0d-5518-438f-858a-77a424b8dd4c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.761648 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf17a32f-809e-4273-ac73-17a8c358bb29-image-registry-private-configuration\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.761648 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:35.761450    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:35.761648 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761471    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9x47\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-kube-api-access-k9x47\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.761648 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:35.761519    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls podName:71be3c38-364e-4ace-8643-1324c5176e0d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:36.261499124 +0000 UTC m=+36.099597693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls") pod "dns-default-lx7wv" (UID: "71be3c38-364e-4ace-8643-1324c5176e0d") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:35.761648 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0314ce0d-5518-438f-858a-77a424b8dd4c-hub\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.761648 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.761648 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0314ce0d-5518-438f-858a-77a424b8dd4c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.761648 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9c5b7fd4-f72c-4e24-a6c0-19b1182875ce-klusterlet-config\") pod \"klusterlet-addon-workmgr-5674f48f96-47stz\" (UID: \"9c5b7fd4-f72c-4e24-a6c0-19b1182875ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:35.761648 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf17a32f-809e-4273-ac73-17a8c358bb29-trusted-ca\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqqnj\" (UniqueName: \"kubernetes.io/projected/71be3c38-364e-4ace-8643-1324c5176e0d-kube-api-access-xqqnj\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:35.761695    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:35.761716    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9f447655-c7crr: secret "image-registry-tls" not found
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:35.761777    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls podName:cf17a32f-809e-4273-ac73-17a8c358bb29 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:36.261758799 +0000 UTC m=+36.099857363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls") pod "image-registry-5c9f447655-c7crr" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29") : secret "image-registry-tls" not found
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf17a32f-809e-4273-ac73-17a8c358bb29-ca-trust-extracted\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c5b7fd4-f72c-4e24-a6c0-19b1182875ce-tmp\") pod \"klusterlet-addon-workmgr-5674f48f96-47stz\" (UID: \"9c5b7fd4-f72c-4e24-a6c0-19b1182875ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5-cert\") pod \"ingress-canary-bzw96\" (UID: \"dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5\") " pod="openshift-ingress-canary/ingress-canary-bzw96"
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s7rl\" (UniqueName: \"kubernetes.io/projected/8c786270-e174-406a-a50c-9672bad97bf2-kube-api-access-4s7rl\") pod \"managed-serviceaccount-addon-agent-79c996d7f8-97xdm\" (UID: \"8c786270-e174-406a-a50c-9672bad97bf2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm"
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4tph\" (UniqueName: \"kubernetes.io/projected/9c5b7fd4-f72c-4e24-a6c0-19b1182875ce-kube-api-access-k4tph\") pod \"klusterlet-addon-workmgr-5674f48f96-47stz\" (UID: \"9c5b7fd4-f72c-4e24-a6c0-19b1182875ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761908    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c786270-e174-406a-a50c-9672bad97bf2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79c996d7f8-97xdm\" (UID: \"8c786270-e174-406a-a50c-9672bad97bf2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm"
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-certificates\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761960    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-bound-sa-token\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.761985    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71be3c38-364e-4ace-8643-1324c5176e0d-tmp-dir\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:35.762286 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.762022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf17a32f-809e-4273-ac73-17a8c358bb29-ca-trust-extracted\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.762937 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.762351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71be3c38-364e-4ace-8643-1324c5176e0d-tmp-dir\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:35.762937 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.762846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf17a32f-809e-4273-ac73-17a8c358bb29-trusted-ca\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.763866 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.763839    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-certificates\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.763866 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.763859    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0314ce0d-5518-438f-858a-77a424b8dd4c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.766969 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.766943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0314ce0d-5518-438f-858a-77a424b8dd4c-ca\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.767077 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.766943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf17a32f-809e-4273-ac73-17a8c358bb29-image-registry-private-configuration\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.767206 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.767184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0314ce0d-5518-438f-858a-77a424b8dd4c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.767728 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.767707    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0314ce0d-5518-438f-858a-77a424b8dd4c-hub\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.767813 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.767709    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf17a32f-809e-4273-ac73-17a8c358bb29-installation-pull-secrets\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.767813 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.767772    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9c5b7fd4-f72c-4e24-a6c0-19b1182875ce-klusterlet-config\") pod \"klusterlet-addon-workmgr-5674f48f96-47stz\" (UID: \"9c5b7fd4-f72c-4e24-a6c0-19b1182875ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:35.767917 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.767813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c786270-e174-406a-a50c-9672bad97bf2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79c996d7f8-97xdm\" (UID: \"8c786270-e174-406a-a50c-9672bad97bf2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm"
Jul 20 09:57:35.768037 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.768020    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0314ce0d-5518-438f-858a-77a424b8dd4c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.768712 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.768694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5-cert\") pod \"ingress-canary-bzw96\" (UID: \"dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5\") " pod="openshift-ingress-canary/ingress-canary-bzw96"
Jul 20 09:57:35.775715 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.775629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9x47\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-kube-api-access-k9x47\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.776758 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.776707    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s7rl\" (UniqueName: \"kubernetes.io/projected/8c786270-e174-406a-a50c-9672bad97bf2-kube-api-access-4s7rl\") pod \"managed-serviceaccount-addon-agent-79c996d7f8-97xdm\" (UID: \"8c786270-e174-406a-a50c-9672bad97bf2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm"
Jul 20 09:57:35.777811 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.777792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp9m5\" (UniqueName: \"kubernetes.io/projected/0314ce0d-5518-438f-858a-77a424b8dd4c-kube-api-access-rp9m5\") pod \"cluster-proxy-proxy-agent-7fc6b68c78-5whmv\" (UID: \"0314ce0d-5518-438f-858a-77a424b8dd4c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.777811 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.777800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-bound-sa-token\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:35.778451 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.778429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqqnj\" (UniqueName: \"kubernetes.io/projected/71be3c38-364e-4ace-8643-1324c5176e0d-kube-api-access-xqqnj\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:35.781441 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.781406    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4tph\" (UniqueName: \"kubernetes.io/projected/9c5b7fd4-f72c-4e24-a6c0-19b1182875ce-kube-api-access-k4tph\") pod \"klusterlet-addon-workmgr-5674f48f96-47stz\" (UID: \"9c5b7fd4-f72c-4e24-a6c0-19b1182875ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:35.849444 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.849397    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm"
Jul 20 09:57:35.859044 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.859024    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:57:35.877647 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:35.877627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:36.267506 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.266801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:36.267506 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.266917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:36.267506 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:36.267055    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:36.267506 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:36.267134    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls podName:71be3c38-364e-4ace-8643-1324c5176e0d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.267096572 +0000 UTC m=+37.105195150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls") pod "dns-default-lx7wv" (UID: "71be3c38-364e-4ace-8643-1324c5176e0d") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:36.267877 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:36.267534    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:36.267877 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:36.267554    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9f447655-c7crr: secret "image-registry-tls" not found
Jul 20 09:57:36.267877 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:36.267610    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls podName:cf17a32f-809e-4273-ac73-17a8c358bb29 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.26759249 +0000 UTC m=+37.105691053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls") pod "image-registry-5c9f447655-c7crr" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29") : secret "image-registry-tls" not found
Jul 20 09:57:36.367697 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.367668    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm"]
Jul 20 09:57:36.373462 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:36.373427    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c786270_e174_406a_a50c_9672bad97bf2.slice/crio-95586691d1bc39f5118a8c53cb3d834f3c97326512907871e6a6bd40587e7b72 WatchSource:0}: Error finding container 95586691d1bc39f5118a8c53cb3d834f3c97326512907871e6a6bd40587e7b72: Status 404 returned error can't find the container with id 95586691d1bc39f5118a8c53cb3d834f3c97326512907871e6a6bd40587e7b72
Jul 20 09:57:36.376049 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.375979    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"]
Jul 20 09:57:36.376891 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.376869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"]
Jul 20 09:57:36.376990 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:36.376971    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0314ce0d_5518_438f_858a_77a424b8dd4c.slice/crio-4f1ee798f980d28241c858c7f95ed8cf35eb677733915700464e690e7c952e90 WatchSource:0}: Error finding container 4f1ee798f980d28241c858c7f95ed8cf35eb677733915700464e690e7c952e90: Status 404 returned error can't find the container with id 4f1ee798f980d28241c858c7f95ed8cf35eb677733915700464e690e7c952e90
Jul 20 09:57:36.379727 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:36.378417    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5b7fd4_f72c_4e24_a6c0_19b1182875ce.slice/crio-51c64cc59b318c392ceac7cb3ca1be29bb4fd2282ed311f9288c546211e9540c WatchSource:0}: Error finding container 51c64cc59b318c392ceac7cb3ca1be29bb4fd2282ed311f9288c546211e9540c: Status 404 returned error can't find the container with id 51c64cc59b318c392ceac7cb3ca1be29bb4fd2282ed311f9288c546211e9540c
Jul 20 09:57:36.421259 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.421229    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 20 09:57:36.422491 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.422474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71be3c38-364e-4ace-8643-1324c5176e0d-config-volume\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:36.624555 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.624531    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mjpn5\""
Jul 20 09:57:36.761316 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.761250    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:57:36.761420 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.761259    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:57:36.761482 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.761258    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:57:36.764107 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.764085    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 09:57:36.764225 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.764151    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fk8qt\""
Jul 20 09:57:36.764225 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.764193    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q8hvk\""
Jul 20 09:57:36.764339 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.764151    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 09:57:36.764572 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.764556    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 09:57:36.764655 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.764619    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 09:57:36.775349 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:36.775331    2561 projected.go:291] Couldn't get configMap openshift-ingress-canary/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Jul 20 09:57:36.816079 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.816063    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 20 09:57:36.825421 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:36.825392    2561 projected.go:196] Error preparing data for projected volume kube-api-access-jsz5l for pod openshift-ingress-canary/ingress-canary-bzw96: failed to sync configmap cache: timed out waiting for the condition
Jul 20 09:57:36.825506 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:36.825458    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5-kube-api-access-jsz5l podName:dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.325436965 +0000 UTC m=+37.163535541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jsz5l" (UniqueName: "kubernetes.io/projected/dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5-kube-api-access-jsz5l") pod "ingress-canary-bzw96" (UID: "dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5") : failed to sync configmap cache: timed out waiting for the condition
Jul 20 09:57:36.948289 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.948251    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv" event={"ID":"0314ce0d-5518-438f-858a-77a424b8dd4c","Type":"ContainerStarted","Data":"4f1ee798f980d28241c858c7f95ed8cf35eb677733915700464e690e7c952e90"}
Jul 20 09:57:36.949870 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.949840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz" event={"ID":"9c5b7fd4-f72c-4e24-a6c0-19b1182875ce","Type":"ContainerStarted","Data":"51c64cc59b318c392ceac7cb3ca1be29bb4fd2282ed311f9288c546211e9540c"}
Jul 20 09:57:36.951438 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.951406    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm" event={"ID":"8c786270-e174-406a-a50c-9672bad97bf2","Type":"ContainerStarted","Data":"95586691d1bc39f5118a8c53cb3d834f3c97326512907871e6a6bd40587e7b72"}
Jul 20 09:57:36.955352 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.955325    2561 generic.go:356] "Generic (PLEG): container finished" podID="102e4f9f-8d5c-40d6-8b00-7f1a13888508" containerID="85e8e1ea06fb535790bfa65653c216afab158af513bcc5d10d3fcc1e8369f107" exitCode=0
Jul 20 09:57:36.955446 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:36.955364    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8s76" event={"ID":"102e4f9f-8d5c-40d6-8b00-7f1a13888508","Type":"ContainerDied","Data":"85e8e1ea06fb535790bfa65653c216afab158af513bcc5d10d3fcc1e8369f107"}
Jul 20 09:57:37.031809 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:37.031730    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 20 09:57:37.276512 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:37.276157    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:37.276512 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:37.276281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:37.276512 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:37.276349    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:37.276512 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:37.276413    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:37.276512 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:37.276421    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls podName:71be3c38-364e-4ace-8643-1324c5176e0d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:39.276400958 +0000 UTC m=+39.114499536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls") pod "dns-default-lx7wv" (UID: "71be3c38-364e-4ace-8643-1324c5176e0d") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:37.276512 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:37.276427    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9f447655-c7crr: secret "image-registry-tls" not found
Jul 20 09:57:37.276512 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:37.276484    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls podName:cf17a32f-809e-4273-ac73-17a8c358bb29 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:39.27646711 +0000 UTC m=+39.114565678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls") pod "image-registry-5c9f447655-c7crr" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29") : secret "image-registry-tls" not found
Jul 20 09:57:37.377449 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:37.377406    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsz5l\" (UniqueName: \"kubernetes.io/projected/dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5-kube-api-access-jsz5l\") pod \"ingress-canary-bzw96\" (UID: \"dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5\") " pod="openshift-ingress-canary/ingress-canary-bzw96"
Jul 20 09:57:37.392151 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:37.392070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsz5l\" (UniqueName: \"kubernetes.io/projected/dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5-kube-api-access-jsz5l\") pod \"ingress-canary-bzw96\" (UID: \"dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5\") " pod="openshift-ingress-canary/ingress-canary-bzw96"
Jul 20 09:57:37.394905 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:37.394718    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bzw96"
Jul 20 09:57:37.608321 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:37.608252    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bzw96"]
Jul 20 09:57:37.613507 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:57:37.613476    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc4c661_d4f8_44b5_8536_5d91d0c7d5a5.slice/crio-b8e6e33502288ccdf319f43d290a6109653548c5a44566700fe13ac37d751bcf WatchSource:0}: Error finding container b8e6e33502288ccdf319f43d290a6109653548c5a44566700fe13ac37d751bcf: Status 404 returned error can't find the container with id b8e6e33502288ccdf319f43d290a6109653548c5a44566700fe13ac37d751bcf
Jul 20 09:57:37.972372 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:37.971184    2561 generic.go:356] "Generic (PLEG): container finished" podID="102e4f9f-8d5c-40d6-8b00-7f1a13888508" containerID="52a5f0fd11c12727c18a40677c5b38310c7b8d37bf5f9b270e4f1aa5a8254c78" exitCode=0
Jul 20 09:57:37.972372 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:37.971505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8s76" event={"ID":"102e4f9f-8d5c-40d6-8b00-7f1a13888508","Type":"ContainerDied","Data":"52a5f0fd11c12727c18a40677c5b38310c7b8d37bf5f9b270e4f1aa5a8254c78"}
Jul 20 09:57:37.986295 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:37.985388    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bzw96" event={"ID":"dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5","Type":"ContainerStarted","Data":"b8e6e33502288ccdf319f43d290a6109653548c5a44566700fe13ac37d751bcf"}
Jul 20 09:57:38.994230 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:38.994194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8s76" event={"ID":"102e4f9f-8d5c-40d6-8b00-7f1a13888508","Type":"ContainerStarted","Data":"b0a4a9672af9cb38e72fd1795db560dc7953f2d9cf96769a2a6ba085aff64645"}
Jul 20 09:57:39.023703 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:39.023641    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n8s76" podStartSLOduration=8.127295458 podStartE2EDuration="38.023579761s" totalImagesPullingTime="29.896284303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:39.023185498 +0000 UTC m=+38.861284099" watchObservedRunningTime="2026-07-20 09:57:39.023579761 +0000 UTC m=+38.861678349"
Jul 20 09:57:39.298025 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:39.297205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:39.298025 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:39.297287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:39.298025 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:39.297470    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:39.298025 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:39.297486    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9f447655-c7crr: secret "image-registry-tls" not found
Jul 20 09:57:39.298025 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:39.297546    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls podName:cf17a32f-809e-4273-ac73-17a8c358bb29 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:43.297526972 +0000 UTC m=+43.135625553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls") pod "image-registry-5c9f447655-c7crr" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29") : secret "image-registry-tls" not found
Jul 20 09:57:39.298025 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:39.297942    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:39.298025 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:39.297991    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls podName:71be3c38-364e-4ace-8643-1324c5176e0d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:43.297975591 +0000 UTC m=+43.136074159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls") pod "dns-default-lx7wv" (UID: "71be3c38-364e-4ace-8643-1324c5176e0d") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:43.335650 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:43.335603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:43.336328 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:43.335709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:43.336328 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:43.335765    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:43.336328 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:43.335782    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9f447655-c7crr: secret "image-registry-tls" not found
Jul 20 09:57:43.336328 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:43.335794    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:43.336328 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:43.335851    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls podName:cf17a32f-809e-4273-ac73-17a8c358bb29 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:51.335829441 +0000 UTC m=+51.173928027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls") pod "image-registry-5c9f447655-c7crr" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29") : secret "image-registry-tls" not found
Jul 20 09:57:43.336328 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:43.335869    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls podName:71be3c38-364e-4ace-8643-1324c5176e0d nodeName:}" failed. No retries permitted until 2026-07-20 09:57:51.335862315 +0000 UTC m=+51.173960891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls") pod "dns-default-lx7wv" (UID: "71be3c38-364e-4ace-8643-1324c5176e0d") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:45.012503 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:45.012475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv" event={"ID":"0314ce0d-5518-438f-858a-77a424b8dd4c","Type":"ContainerStarted","Data":"833d4b15369bb4309540824fb739b5acb42061e557b00dd022de67edf3c98ce4"}
Jul 20 09:57:45.012904 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:45.012515    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv" event={"ID":"0314ce0d-5518-438f-858a-77a424b8dd4c","Type":"ContainerStarted","Data":"3ebc0b47ca5834af5508521b863349a50986dea1806203e13c29ca9f25a05fa5"}
Jul 20 09:57:45.012904 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:45.012531    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv" event={"ID":"0314ce0d-5518-438f-858a-77a424b8dd4c","Type":"ContainerStarted","Data":"daedf4b3a0d1db50159d9c65e410b4b1530fe23e9a6578e6e2355825ac9ac5db"}
Jul 20 09:57:45.013994 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:45.013971    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm" event={"ID":"8c786270-e174-406a-a50c-9672bad97bf2","Type":"ContainerStarted","Data":"ad45ebeff156bee37fdc73192a4c3778149d7c0cb34ea62ac882da8c5bad387b"}
Jul 20 09:57:45.015376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:45.015356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bzw96" event={"ID":"dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5","Type":"ContainerStarted","Data":"41e6bcdcfb3b10b01dd475f9eb4e0f036ce893b5f07f017864b0a8afc7003ad1"}
Jul 20 09:57:45.016668 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:45.016646    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz" event={"ID":"9c5b7fd4-f72c-4e24-a6c0-19b1182875ce","Type":"ContainerStarted","Data":"eed51a03a4b07ac606bb1ccf6e573a108e4ea27adbc9a8b0d8dfd1d562368424"}
Jul 20 09:57:45.016855 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:45.016839    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:45.018447 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:45.018431    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz"
Jul 20 09:57:45.035359 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:45.035320    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv" podStartSLOduration=21.109651191 podStartE2EDuration="29.035306108s" totalImagesPullingTime="7.925654917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:45.034796322 +0000 UTC m=+44.872894906" watchObservedRunningTime="2026-07-20 09:57:45.035306108 +0000 UTC m=+44.873404694"
Jul 20 09:57:45.056051 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:45.056014    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c996d7f8-97xdm" podStartSLOduration=21.143061262 podStartE2EDuration="29.056003057s" totalImagesPullingTime="7.912941795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:45.055920547 +0000 UTC m=+44.894019148" watchObservedRunningTime="2026-07-20 09:57:45.056003057 +0000 UTC m=+44.894101642"
Jul 20 09:57:45.075439 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:45.075407    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bzw96" podStartSLOduration=3.40300098 podStartE2EDuration="10.075393839s" totalImagesPullingTime="6.672392859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:45.074750836 +0000 UTC m=+44.912849421" watchObservedRunningTime="2026-07-20 09:57:45.075393839 +0000 UTC m=+44.913492421"
Jul 20 09:57:45.097564 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:45.097520    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5674f48f96-47stz" podStartSLOduration=21.191003398 podStartE2EDuration="29.097507099s" totalImagesPullingTime="7.906503701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:45.096070215 +0000 UTC m=+44.934168800" watchObservedRunningTime="2026-07-20 09:57:45.097507099 +0000 UTC m=+44.935605737"
Jul 20 09:57:51.394313 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:51.394280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:57:51.394689 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:57:51.394353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:57:51.394689 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:51.394453    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:51.394689 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:51.394482    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9f447655-c7crr: secret "image-registry-tls" not found
Jul 20 09:57:51.394689 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:51.394457    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:51.394689 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:51.394545    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls podName:71be3c38-364e-4ace-8643-1324c5176e0d nodeName:}" failed. No retries permitted until 2026-07-20 09:58:07.394532451 +0000 UTC m=+67.232631015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls") pod "dns-default-lx7wv" (UID: "71be3c38-364e-4ace-8643-1324c5176e0d") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:51.394689 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:57:51.394557    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls podName:cf17a32f-809e-4273-ac73-17a8c358bb29 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:07.394551188 +0000 UTC m=+67.232649750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls") pod "image-registry-5c9f447655-c7crr" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29") : secret "image-registry-tls" not found
Jul 20 09:58:00.942535 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:00.942501    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxc68"
Jul 20 09:58:06.401771 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.401729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:58:06.402220 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.401796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:58:06.404698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.404676    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 09:58:06.404814 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.404706    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 09:58:06.412837 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:06.412814    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 20 09:58:06.412905 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:06.412895    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs podName:df44767e-e243-4004-a43e-4d2982ddd197 nodeName:}" failed. No retries permitted until 2026-07-20 09:59:10.412875136 +0000 UTC m=+130.250973699 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs") pod "network-metrics-daemon-n4x2n" (UID: "df44767e-e243-4004-a43e-4d2982ddd197") : secret "metrics-daemon-secret" not found
Jul 20 09:58:06.415628 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.415603    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ab13120-6b0f-4805-bc85-bb1ee7384987-original-pull-secret\") pod \"global-pull-secret-syncer-rw9rr\" (UID: \"3ab13120-6b0f-4805-bc85-bb1ee7384987\") " pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:58:06.483917 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.483893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw9rr"
Jul 20 09:58:06.502912 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.502891    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h5s\" (UniqueName: \"kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s\") pod \"network-check-target-fcpp5\" (UID: \"ffd754c6-58d4-429e-aa73-9e3cd0f1626e\") " pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:58:06.505459 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.505435    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 09:58:06.516351 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.516327    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 09:58:06.527185 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.527160    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8h5s\" (UniqueName: \"kubernetes.io/projected/ffd754c6-58d4-429e-aa73-9e3cd0f1626e-kube-api-access-z8h5s\") pod \"network-check-target-fcpp5\" (UID: \"ffd754c6-58d4-429e-aa73-9e3cd0f1626e\") " pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:58:06.600791 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.600759    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rw9rr"]
Jul 20 09:58:06.603947 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:58:06.603922    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab13120_6b0f_4805_bc85_bb1ee7384987.slice/crio-6defa9fa52b1d10832527189cecf5ed6844952a19522c4c35bbeb5cc64b91109 WatchSource:0}: Error finding container 6defa9fa52b1d10832527189cecf5ed6844952a19522c4c35bbeb5cc64b91109: Status 404 returned error can't find the container with id 6defa9fa52b1d10832527189cecf5ed6844952a19522c4c35bbeb5cc64b91109
Jul 20 09:58:06.791137 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.791055    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fk8qt\""
Jul 20 09:58:06.798706 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.798689    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:58:06.914614 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:06.914584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fcpp5"]
Jul 20 09:58:06.917823 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:58:06.917792    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd754c6_58d4_429e_aa73_9e3cd0f1626e.slice/crio-96ccea7065585b10707ad499a8edf6a6753dd15a173a210eb1c3f471f22c2a76 WatchSource:0}: Error finding container 96ccea7065585b10707ad499a8edf6a6753dd15a173a210eb1c3f471f22c2a76: Status 404 returned error can't find the container with id 96ccea7065585b10707ad499a8edf6a6753dd15a173a210eb1c3f471f22c2a76
Jul 20 09:58:07.084340 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:07.084256    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fcpp5" event={"ID":"ffd754c6-58d4-429e-aa73-9e3cd0f1626e","Type":"ContainerStarted","Data":"96ccea7065585b10707ad499a8edf6a6753dd15a173a210eb1c3f471f22c2a76"}
Jul 20 09:58:07.085448 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:07.085408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rw9rr" event={"ID":"3ab13120-6b0f-4805-bc85-bb1ee7384987","Type":"ContainerStarted","Data":"6defa9fa52b1d10832527189cecf5ed6844952a19522c4c35bbeb5cc64b91109"}
Jul 20 09:58:07.410745 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:07.410705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:58:07.411236 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:07.410806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:58:07.411236 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:07.410914    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:58:07.411236 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:07.410931    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:58:07.411236 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:07.410940    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9f447655-c7crr: secret "image-registry-tls" not found
Jul 20 09:58:07.411236 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:07.410993    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls podName:71be3c38-364e-4ace-8643-1324c5176e0d nodeName:}" failed. No retries permitted until 2026-07-20 09:58:39.410973987 +0000 UTC m=+99.249072555 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls") pod "dns-default-lx7wv" (UID: "71be3c38-364e-4ace-8643-1324c5176e0d") : secret "dns-default-metrics-tls" not found
Jul 20 09:58:07.411236 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:07.411011    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls podName:cf17a32f-809e-4273-ac73-17a8c358bb29 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:39.411002423 +0000 UTC m=+99.249100991 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls") pod "image-registry-5c9f447655-c7crr" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29") : secret "image-registry-tls" not found
Jul 20 09:58:12.103092 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:12.103057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fcpp5" event={"ID":"ffd754c6-58d4-429e-aa73-9e3cd0f1626e","Type":"ContainerStarted","Data":"cd4585c2328a0d6d694a3ab4289f1828e2dd8d4609b14e01f6be0b4e02e52357"}
Jul 20 09:58:12.103565 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:12.103165    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:58:12.104511 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:12.104489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rw9rr" event={"ID":"3ab13120-6b0f-4805-bc85-bb1ee7384987","Type":"ContainerStarted","Data":"2717c01bbbe41bfc15d0c4210cfa0c95595810351ce0fbcd27371cd6a4598dea"}
Jul 20 09:58:12.123351 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:12.123312    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fcpp5" podStartSLOduration=66.833621149 podStartE2EDuration="1m11.123298433s" totalImagesPullingTime="4.289677284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:12.122809888 +0000 UTC m=+71.960908474" watchObservedRunningTime="2026-07-20 09:58:12.123298433 +0000 UTC m=+71.961396996"
Jul 20 09:58:12.139311 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:12.139270    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rw9rr" podStartSLOduration=66.532187302 podStartE2EDuration="1m11.139259073s" totalImagesPullingTime="4.607071771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:12.139052884 +0000 UTC m=+71.977151479" watchObservedRunningTime="2026-07-20 09:58:12.139259073 +0000 UTC m=+71.977357658"
Jul 20 09:58:27.606739 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.606596    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ngrqc"]
Jul 20 09:58:27.609035 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.609020    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.611837 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.611818    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 20 09:58:27.615096 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.614670    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 20 09:58:27.615096 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.614674    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 20 09:58:27.615096 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.614748    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 20 09:58:27.615096 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.614914    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pkkvc\""
Jul 20 09:58:27.629684 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.629660    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ngrqc"]
Jul 20 09:58:27.754365 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.754339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.754365 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.754370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbwm5\" (UniqueName: \"kubernetes.io/projected/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-api-access-wbwm5\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.754546 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.754449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-data-volume\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.754546 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.754491    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.754546 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.754527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-crio-socket\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.855537 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.855500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-data-volume\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.855657 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.855548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.855657 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:27.855631    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:27.855725 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.855679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-crio-socket\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.855725 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:27.855695    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls podName:ce4ca6d7-070c-410b-8c44-ccc8b089bca8 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:28.355679902 +0000 UTC m=+88.193778472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngrqc" (UID: "ce4ca6d7-070c-410b-8c44-ccc8b089bca8") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:27.855725 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.855721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.855824 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.855731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-crio-socket\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.855824 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.855740    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbwm5\" (UniqueName: \"kubernetes.io/projected/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-api-access-wbwm5\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.855824 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:27.855781    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:27.855824 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.855815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-data-volume\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:27.855941 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:27.855831    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm podName:ce4ca6d7-070c-410b-8c44-ccc8b089bca8 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:28.355820017 +0000 UTC m=+88.193918602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ngrqc" (UID: "ce4ca6d7-070c-410b-8c44-ccc8b089bca8") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:27.866407 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:27.866386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbwm5\" (UniqueName: \"kubernetes.io/projected/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-api-access-wbwm5\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:28.358342 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:28.358250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:28.358505 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:28.358345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:28.358505 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:28.358381    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:28.358505 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:28.358451    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm podName:ce4ca6d7-070c-410b-8c44-ccc8b089bca8 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:29.358435278 +0000 UTC m=+89.196533840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ngrqc" (UID: "ce4ca6d7-070c-410b-8c44-ccc8b089bca8") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:28.358505 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:28.358452    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:28.358505 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:28.358502    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls podName:ce4ca6d7-070c-410b-8c44-ccc8b089bca8 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:29.358486654 +0000 UTC m=+89.196585217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngrqc" (UID: "ce4ca6d7-070c-410b-8c44-ccc8b089bca8") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:29.360967 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:29.360937    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mn8bv_6a052b9a-0d1c-4b01-8b75-1f435ec83841/dns-node-resolver/0.log"
Jul 20 09:58:29.368234 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:29.368213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:29.368315 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:29.368265    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:29.368370 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:29.368332    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:29.368419 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:29.368378    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm podName:ce4ca6d7-070c-410b-8c44-ccc8b089bca8 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:31.368365673 +0000 UTC m=+91.206464235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ngrqc" (UID: "ce4ca6d7-070c-410b-8c44-ccc8b089bca8") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:29.368419 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:29.368384    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:29.368527 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:29.368463    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls podName:ce4ca6d7-070c-410b-8c44-ccc8b089bca8 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:31.368444343 +0000 UTC m=+91.206542909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngrqc" (UID: "ce4ca6d7-070c-410b-8c44-ccc8b089bca8") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:30.561825 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:30.561795    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n5hmv_33f1203a-dd9a-46be-96bd-47184ae3fccf/node-ca/0.log"
Jul 20 09:58:30.960024 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:30.959996    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bzw96_dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5/serve-healthcheck-canary/0.log"
Jul 20 09:58:31.383836 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:31.383800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:31.384018 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:31.383890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:31.384018 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:31.383927    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:31.384018 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:31.383997    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm podName:ce4ca6d7-070c-410b-8c44-ccc8b089bca8 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:35.38398027 +0000 UTC m=+95.222078835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ngrqc" (UID: "ce4ca6d7-070c-410b-8c44-ccc8b089bca8") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:31.384204 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:31.384032    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:31.384204 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:31.384092    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls podName:ce4ca6d7-070c-410b-8c44-ccc8b089bca8 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:35.384076169 +0000 UTC m=+95.222174740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngrqc" (UID: "ce4ca6d7-070c-410b-8c44-ccc8b089bca8") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:35.417894 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:35.417856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:35.418419 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:35.417913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:35.418419 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:35.417979    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:35.418419 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:35.418029    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm podName:ce4ca6d7-070c-410b-8c44-ccc8b089bca8 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:43.418015821 +0000 UTC m=+103.256114384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ngrqc" (UID: "ce4ca6d7-070c-410b-8c44-ccc8b089bca8") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:35.418419 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:35.418027    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:35.418419 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:35.418105    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls podName:ce4ca6d7-070c-410b-8c44-ccc8b089bca8 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:43.418086491 +0000 UTC m=+103.256185061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngrqc" (UID: "ce4ca6d7-070c-410b-8c44-ccc8b089bca8") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:39.447831 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:39.447797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls\") pod \"image-registry-5c9f447655-c7crr\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") " pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:58:39.448228 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:39.447866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:58:39.448228 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:39.447961    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:58:39.448228 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:39.447975    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:58:39.448228 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:39.447983    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9f447655-c7crr: secret "image-registry-tls" not found
Jul 20 09:58:39.448228 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:39.448041    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls podName:cf17a32f-809e-4273-ac73-17a8c358bb29 nodeName:}" failed. No retries permitted until 2026-07-20 09:59:43.448019402 +0000 UTC m=+163.286117973 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls") pod "image-registry-5c9f447655-c7crr" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29") : secret "image-registry-tls" not found
Jul 20 09:58:39.448228 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:39.448060    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls podName:71be3c38-364e-4ace-8643-1324c5176e0d nodeName:}" failed. No retries permitted until 2026-07-20 09:59:43.448051992 +0000 UTC m=+163.286150558 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls") pod "dns-default-lx7wv" (UID: "71be3c38-364e-4ace-8643-1324c5176e0d") : secret "dns-default-metrics-tls" not found
Jul 20 09:58:43.109236 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:43.109198    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fcpp5"
Jul 20 09:58:43.476637 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:43.476605    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:43.476800 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:43.476661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:43.476800 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:43.476746    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:43.476883 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:58:43.476811    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls podName:ce4ca6d7-070c-410b-8c44-ccc8b089bca8 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:59.476795567 +0000 UTC m=+119.314894130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngrqc" (UID: "ce4ca6d7-070c-410b-8c44-ccc8b089bca8") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:43.477104 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:43.477087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:55.860912 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:55.860869    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv" podUID="0314ce0d-5518-438f-858a-77a424b8dd4c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 20 09:58:59.491869 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:59.491828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:59.494043 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:59.494023    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce4ca6d7-070c-410b-8c44-ccc8b089bca8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngrqc\" (UID: \"ce4ca6d7-070c-410b-8c44-ccc8b089bca8\") " pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:59.717487 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:59.717456    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ngrqc"
Jul 20 09:58:59.836273 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:58:59.836238    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ngrqc"]
Jul 20 09:58:59.839520 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:58:59.839495    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4ca6d7_070c_410b_8c44_ccc8b089bca8.slice/crio-ec232154d276748327fe42ea7be0c1e7b849a95ddfd523ec913b99144201e730 WatchSource:0}: Error finding container ec232154d276748327fe42ea7be0c1e7b849a95ddfd523ec913b99144201e730: Status 404 returned error can't find the container with id ec232154d276748327fe42ea7be0c1e7b849a95ddfd523ec913b99144201e730
Jul 20 09:59:00.237167 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:00.237132    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngrqc" event={"ID":"ce4ca6d7-070c-410b-8c44-ccc8b089bca8","Type":"ContainerStarted","Data":"ec232154d276748327fe42ea7be0c1e7b849a95ddfd523ec913b99144201e730"}
Jul 20 09:59:05.256850 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:05.256807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngrqc" event={"ID":"ce4ca6d7-070c-410b-8c44-ccc8b089bca8","Type":"ContainerStarted","Data":"bb9a9ee3c0a8776813457169284962a2a538978768e004fac36f3fb165ec406e"}
Jul 20 09:59:05.860613 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:05.860570    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv" podUID="0314ce0d-5518-438f-858a-77a424b8dd4c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 20 09:59:08.268925 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:08.268847    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngrqc" event={"ID":"ce4ca6d7-070c-410b-8c44-ccc8b089bca8","Type":"ContainerStarted","Data":"8c6f59e44eaa73d13c4daa108fbfab45cdffb3cfa74ecc5bd91bb0ae77222fa8"}
Jul 20 09:59:10.484293 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:10.484252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:59:10.486742 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:10.486713    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df44767e-e243-4004-a43e-4d2982ddd197-metrics-certs\") pod \"network-metrics-daemon-n4x2n\" (UID: \"df44767e-e243-4004-a43e-4d2982ddd197\") " pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:59:10.679952 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:10.679917    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q8hvk\""
Jul 20 09:59:10.687412 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:10.687386    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4x2n"
Jul 20 09:59:11.435442 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:11.435418    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n4x2n"]
Jul 20 09:59:11.437926 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:59:11.437895    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf44767e_e243_4004_a43e_4d2982ddd197.slice/crio-9d2f7a7e5bbe56f3461849a7fe6d4508bf1de845759056039f9fb8eeae169aef WatchSource:0}: Error finding container 9d2f7a7e5bbe56f3461849a7fe6d4508bf1de845759056039f9fb8eeae169aef: Status 404 returned error can't find the container with id 9d2f7a7e5bbe56f3461849a7fe6d4508bf1de845759056039f9fb8eeae169aef
Jul 20 09:59:12.284054 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:12.284023    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngrqc" event={"ID":"ce4ca6d7-070c-410b-8c44-ccc8b089bca8","Type":"ContainerStarted","Data":"6471ab2f7f2a04f839cf42b791a40262216f6d57c1eaefa8c785c8b393016430"}
Jul 20 09:59:12.285100 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:12.285078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n4x2n" event={"ID":"df44767e-e243-4004-a43e-4d2982ddd197","Type":"ContainerStarted","Data":"9d2f7a7e5bbe56f3461849a7fe6d4508bf1de845759056039f9fb8eeae169aef"}
Jul 20 09:59:12.307837 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:12.307797    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ngrqc" podStartSLOduration=33.886096562 podStartE2EDuration="45.30778279s" totalImagesPullingTime="11.421686228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:58:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:12.306283622 +0000 UTC m=+132.144382228" watchObservedRunningTime="2026-07-20 09:59:12.30778279 +0000 UTC m=+132.145881375"
Jul 20 09:59:13.290268 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:13.290230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n4x2n" event={"ID":"df44767e-e243-4004-a43e-4d2982ddd197","Type":"ContainerStarted","Data":"c0a83a1c48a38a4e123dcaa5036c19b3f1ecf481d63dc6325eaf07807383f0a5"}
Jul 20 09:59:13.290674 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:13.290286    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n4x2n" event={"ID":"df44767e-e243-4004-a43e-4d2982ddd197","Type":"ContainerStarted","Data":"fbe3bdfb43f61fc2290983e97cd4cd1368c8067fa6b882954fe612b7eb57d2c8"}
Jul 20 09:59:13.307370 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:13.307321    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n4x2n" podStartSLOduration=132.465587159 podStartE2EDuration="2m13.30730642s" totalImagesPullingTime="841.719261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:13.306052956 +0000 UTC m=+133.144151542" watchObservedRunningTime="2026-07-20 09:59:13.30730642 +0000 UTC m=+133.145405004"
Jul 20 09:59:15.860516 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:15.860480    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv" podUID="0314ce0d-5518-438f-858a-77a424b8dd4c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 20 09:59:15.860894 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:15.860551    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv"
Jul 20 09:59:15.861000 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:15.860972    2561 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"833d4b15369bb4309540824fb739b5acb42061e557b00dd022de67edf3c98ce4"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 20 09:59:15.861036 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:15.861021    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv" podUID="0314ce0d-5518-438f-858a-77a424b8dd4c" containerName="service-proxy" containerID="cri-o://833d4b15369bb4309540824fb739b5acb42061e557b00dd022de67edf3c98ce4" gracePeriod=30
Jul 20 09:59:16.301921 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:16.301838    2561 generic.go:356] "Generic (PLEG): container finished" podID="0314ce0d-5518-438f-858a-77a424b8dd4c" containerID="833d4b15369bb4309540824fb739b5acb42061e557b00dd022de67edf3c98ce4" exitCode=2
Jul 20 09:59:16.301921 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:16.301884    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv" event={"ID":"0314ce0d-5518-438f-858a-77a424b8dd4c","Type":"ContainerDied","Data":"833d4b15369bb4309540824fb739b5acb42061e557b00dd022de67edf3c98ce4"}
Jul 20 09:59:16.301921 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:16.301917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fc6b68c78-5whmv" event={"ID":"0314ce0d-5518-438f-858a-77a424b8dd4c","Type":"ContainerStarted","Data":"348e9560675338ae2a27769e6b25a41f524c0517c5faacb7f318fd910953e604"}
Jul 20 09:59:25.704305 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.704271    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tqkhq"]
Jul 20 09:59:25.707316 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.707299    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.710733 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.710711    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 20 09:59:25.711101 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.711086    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 20 09:59:25.711293 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.711276    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g7gtr\""
Jul 20 09:59:25.711829 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.711811    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 20 09:59:25.712296 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.712277    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 20 09:59:25.712415 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.712313    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 20 09:59:25.713046 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.713031    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 20 09:59:25.788828 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.788800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-textfile\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.788828 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.788829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c096253-3ee3-424e-bbfe-40d8360c181a-metrics-client-ca\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.788977 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.788848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c096253-3ee3-424e-bbfe-40d8360c181a-root\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.788977 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.788865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.788977 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.788940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c096253-3ee3-424e-bbfe-40d8360c181a-sys\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.788977 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.788967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-wtmp\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.789094 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.789014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.789094 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.789032    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgw9\" (UniqueName: \"kubernetes.io/projected/5c096253-3ee3-424e-bbfe-40d8360c181a-kube-api-access-7mgw9\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.789094 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.789062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-tls\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889302 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889403 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgw9\" (UniqueName: \"kubernetes.io/projected/5c096253-3ee3-424e-bbfe-40d8360c181a-kube-api-access-7mgw9\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889403 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-tls\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889493 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-textfile\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889493 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c096253-3ee3-424e-bbfe-40d8360c181a-metrics-client-ca\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889493 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c096253-3ee3-424e-bbfe-40d8360c181a-root\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889601 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c096253-3ee3-424e-bbfe-40d8360c181a-root\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889601 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:59:25.889522    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 20 09:59:25.889601 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889601 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:59:25.889596    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-tls podName:5c096253-3ee3-424e-bbfe-40d8360c181a nodeName:}" failed. No retries permitted until 2026-07-20 09:59:26.3895755 +0000 UTC m=+146.227674063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-tls") pod "node-exporter-tqkhq" (UID: "5c096253-3ee3-424e-bbfe-40d8360c181a") : secret "node-exporter-tls" not found
Jul 20 09:59:25.889794 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c096253-3ee3-424e-bbfe-40d8360c181a-sys\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889794 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889643    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-wtmp\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889794 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c096253-3ee3-424e-bbfe-40d8360c181a-sys\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889906 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-wtmp\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.889906 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.889850    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-textfile\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.890076 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.890060    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.890076 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.890071    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c096253-3ee3-424e-bbfe-40d8360c181a-metrics-client-ca\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.891658 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.891638    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:25.907386 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:25.907363    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgw9\" (UniqueName: \"kubernetes.io/projected/5c096253-3ee3-424e-bbfe-40d8360c181a-kube-api-access-7mgw9\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:26.393303 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:26.393273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-tls\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:26.395499 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:26.395475    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c096253-3ee3-424e-bbfe-40d8360c181a-node-exporter-tls\") pod \"node-exporter-tqkhq\" (UID: \"5c096253-3ee3-424e-bbfe-40d8360c181a\") " pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:26.422804 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:26.422779    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c9f447655-c7crr"]
Jul 20 09:59:26.423028 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:59:26.423008    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5c9f447655-c7crr" podUID="cf17a32f-809e-4273-ac73-17a8c358bb29"
Jul 20 09:59:26.615749 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:26.615722    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqkhq"
Jul 20 09:59:26.626425 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:59:26.626401    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c096253_3ee3_424e_bbfe_40d8360c181a.slice/crio-db808cb63b5c28b3607e68cffa9243963085a3c6fdaa912c27281b3a276a37f4 WatchSource:0}: Error finding container db808cb63b5c28b3607e68cffa9243963085a3c6fdaa912c27281b3a276a37f4: Status 404 returned error can't find the container with id db808cb63b5c28b3607e68cffa9243963085a3c6fdaa912c27281b3a276a37f4
Jul 20 09:59:27.338469 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.338435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqkhq" event={"ID":"5c096253-3ee3-424e-bbfe-40d8360c181a","Type":"ContainerStarted","Data":"db808cb63b5c28b3607e68cffa9243963085a3c6fdaa912c27281b3a276a37f4"}
Jul 20 09:59:27.338951 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.338442    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:59:27.347420 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.347392    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:59:27.502415 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.502394    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-bound-sa-token\") pod \"cf17a32f-809e-4273-ac73-17a8c358bb29\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") "
Jul 20 09:59:27.502533 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.502432    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf17a32f-809e-4273-ac73-17a8c358bb29-image-registry-private-configuration\") pod \"cf17a32f-809e-4273-ac73-17a8c358bb29\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") "
Jul 20 09:59:27.502533 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.502462    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf17a32f-809e-4273-ac73-17a8c358bb29-installation-pull-secrets\") pod \"cf17a32f-809e-4273-ac73-17a8c358bb29\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") "
Jul 20 09:59:27.502640 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.502580    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9x47\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-kube-api-access-k9x47\") pod \"cf17a32f-809e-4273-ac73-17a8c358bb29\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") "
Jul 20 09:59:27.502698 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.502658    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf17a32f-809e-4273-ac73-17a8c358bb29-trusted-ca\") pod \"cf17a32f-809e-4273-ac73-17a8c358bb29\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") "
Jul 20 09:59:27.502760 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.502715    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf17a32f-809e-4273-ac73-17a8c358bb29-ca-trust-extracted\") pod \"cf17a32f-809e-4273-ac73-17a8c358bb29\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") "
Jul 20 09:59:27.502817 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.502777    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-certificates\") pod \"cf17a32f-809e-4273-ac73-17a8c358bb29\" (UID: \"cf17a32f-809e-4273-ac73-17a8c358bb29\") "
Jul 20 09:59:27.503035 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.503007    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf17a32f-809e-4273-ac73-17a8c358bb29-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cf17a32f-809e-4273-ac73-17a8c358bb29" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 09:59:27.503165 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.503138    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf17a32f-809e-4273-ac73-17a8c358bb29-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cf17a32f-809e-4273-ac73-17a8c358bb29" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:27.503256 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.503233    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cf17a32f-809e-4273-ac73-17a8c358bb29" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:27.504679 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.504643    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf17a32f-809e-4273-ac73-17a8c358bb29-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cf17a32f-809e-4273-ac73-17a8c358bb29" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:27.504842 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.504825    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-kube-api-access-k9x47" (OuterVolumeSpecName: "kube-api-access-k9x47") pod "cf17a32f-809e-4273-ac73-17a8c358bb29" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29"). InnerVolumeSpecName "kube-api-access-k9x47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:27.505005 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.504985    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf17a32f-809e-4273-ac73-17a8c358bb29-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cf17a32f-809e-4273-ac73-17a8c358bb29" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:27.505051 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.505032    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cf17a32f-809e-4273-ac73-17a8c358bb29" (UID: "cf17a32f-809e-4273-ac73-17a8c358bb29"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:27.603465 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.603414    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf17a32f-809e-4273-ac73-17a8c358bb29-ca-trust-extracted\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 09:59:27.603465 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.603436    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-certificates\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 09:59:27.603465 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.603445    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-bound-sa-token\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 09:59:27.603465 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.603453    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf17a32f-809e-4273-ac73-17a8c358bb29-image-registry-private-configuration\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 09:59:27.603465 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.603462    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf17a32f-809e-4273-ac73-17a8c358bb29-installation-pull-secrets\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 09:59:27.603663 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.603471    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9x47\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-kube-api-access-k9x47\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 09:59:27.603663 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:27.603479    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf17a32f-809e-4273-ac73-17a8c358bb29-trusted-ca\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 09:59:28.343201 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:28.343163    2561 generic.go:356] "Generic (PLEG): container finished" podID="5c096253-3ee3-424e-bbfe-40d8360c181a" containerID="4163dfe56474964d0159d3f013c073ae84bdf40e5fbb1ad5dc09d539346b9f62" exitCode=0
Jul 20 09:59:28.343647 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:28.343240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqkhq" event={"ID":"5c096253-3ee3-424e-bbfe-40d8360c181a","Type":"ContainerDied","Data":"4163dfe56474964d0159d3f013c073ae84bdf40e5fbb1ad5dc09d539346b9f62"}
Jul 20 09:59:28.343647 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:28.343271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c9f447655-c7crr"
Jul 20 09:59:28.399217 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:28.399192    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c9f447655-c7crr"]
Jul 20 09:59:28.411527 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:28.411500    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c9f447655-c7crr"]
Jul 20 09:59:28.511136 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:28.511085    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf17a32f-809e-4273-ac73-17a8c358bb29-registry-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 09:59:28.764932 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:28.764901    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf17a32f-809e-4273-ac73-17a8c358bb29" path="/var/lib/kubelet/pods/cf17a32f-809e-4273-ac73-17a8c358bb29/volumes"
Jul 20 09:59:29.348281 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:29.348245    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqkhq" event={"ID":"5c096253-3ee3-424e-bbfe-40d8360c181a","Type":"ContainerStarted","Data":"15f080c4ce1b4bcdc7b7155908f1d141681c0f72c610b18948f564c433cd0f37"}
Jul 20 09:59:29.348670 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:29.348296    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqkhq" event={"ID":"5c096253-3ee3-424e-bbfe-40d8360c181a","Type":"ContainerStarted","Data":"444d6fb9b7d4e0d813079e95bfd8715b15ececb5caa8f959b9f81df65791b655"}
Jul 20 09:59:29.370376 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:29.370318    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tqkhq" podStartSLOduration=3.506589558 podStartE2EDuration="4.370304136s" totalImagesPullingTime="800.014428ms" totalInitContainerRuntime="63.70015ms" isStatefulPod=true podCreationTimestamp="2026-07-20 09:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:29.369598862 +0000 UTC m=+149.207697447" watchObservedRunningTime="2026-07-20 09:59:29.370304136 +0000 UTC m=+149.208402721"
Jul 20 09:59:38.599579 ip-10-0-134-80 kubenswrapper[2561]: E0720 09:59:38.599542    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-lx7wv" podUID="71be3c38-364e-4ace-8643-1324c5176e0d"
Jul 20 09:59:39.378382 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:39.378354    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:59:43.515583 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:43.515547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:59:43.517844 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:43.517822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71be3c38-364e-4ace-8643-1324c5176e0d-metrics-tls\") pod \"dns-default-lx7wv\" (UID: \"71be3c38-364e-4ace-8643-1324c5176e0d\") " pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:59:43.582384 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:43.582361    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7r5mf\""
Jul 20 09:59:43.589392 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:43.589375    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:59:43.752752 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:43.752714    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lx7wv"]
Jul 20 09:59:43.753712 ip-10-0-134-80 kubenswrapper[2561]: W0720 09:59:43.753690    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71be3c38_364e_4ace_8643_1324c5176e0d.slice/crio-122897ddc595c4f7602159efc4244d335716194d3301c456702d5690a85bab63 WatchSource:0}: Error finding container 122897ddc595c4f7602159efc4244d335716194d3301c456702d5690a85bab63: Status 404 returned error can't find the container with id 122897ddc595c4f7602159efc4244d335716194d3301c456702d5690a85bab63
Jul 20 09:59:44.396343 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:44.396305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lx7wv" event={"ID":"71be3c38-364e-4ace-8643-1324c5176e0d","Type":"ContainerStarted","Data":"122897ddc595c4f7602159efc4244d335716194d3301c456702d5690a85bab63"}
Jul 20 09:59:45.401919 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:45.401881    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lx7wv" event={"ID":"71be3c38-364e-4ace-8643-1324c5176e0d","Type":"ContainerStarted","Data":"e7450726c48e52cb8ed08f0bebdc1b0dde45f2cee668ad4a1442114dc610551b"}
Jul 20 09:59:45.402322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:45.401941    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lx7wv" event={"ID":"71be3c38-364e-4ace-8643-1324c5176e0d","Type":"ContainerStarted","Data":"2888ccffccd42f2bec722cdb0aa461297fbc0a620f0a68291c115ee3d72feee8"}
Jul 20 09:59:45.402322 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:45.402002    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lx7wv"
Jul 20 09:59:45.422647 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:45.422605    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lx7wv" podStartSLOduration=128.944029263 podStartE2EDuration="2m10.422587391s" totalImagesPullingTime="1.478558128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:45.421329725 +0000 UTC m=+165.259428354" watchObservedRunningTime="2026-07-20 09:59:45.422587391 +0000 UTC m=+165.260686017"
Jul 20 09:59:55.407801 ip-10-0-134-80 kubenswrapper[2561]: I0720 09:59:55.407729    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lx7wv"
Jul 20 10:02:00.595926 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:02:00.595895    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:02:00.599105 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:02:00.599082    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 20 10:02:00.603915 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:02:00.603896    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:06:00.205759 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.205671    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-48w42"]
Jul 20 10:06:00.208902 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.208886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-48w42"
Jul 20 10:06:00.214560 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.214539    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 20 10:06:00.215647 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.215627    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 20 10:06:00.215647 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.215636    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 20 10:06:00.215795 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.215656    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-zftbl\""
Jul 20 10:06:00.229033 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.229010    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-48w42"]
Jul 20 10:06:00.266775 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.266751    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-pdprk"]
Jul 20 10:06:00.269619 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.269605    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-pdprk"
Jul 20 10:06:00.285879 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.285856    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9hg6l\""
Jul 20 10:06:00.285982 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.285951    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 20 10:06:00.304582 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.304559    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-pdprk"]
Jul 20 10:06:00.322075 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.322047    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76hv9\" (UniqueName: \"kubernetes.io/projected/5fffbf01-d444-4125-aef6-5fdaabad49cc-kube-api-access-76hv9\") pod \"kserve-controller-manager-645f4f9567-48w42\" (UID: \"5fffbf01-d444-4125-aef6-5fdaabad49cc\") " pod="kserve/kserve-controller-manager-645f4f9567-48w42"
Jul 20 10:06:00.322225 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.322098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fffbf01-d444-4125-aef6-5fdaabad49cc-cert\") pod \"kserve-controller-manager-645f4f9567-48w42\" (UID: \"5fffbf01-d444-4125-aef6-5fdaabad49cc\") " pod="kserve/kserve-controller-manager-645f4f9567-48w42"
Jul 20 10:06:00.423374 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.423344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fffbf01-d444-4125-aef6-5fdaabad49cc-cert\") pod \"kserve-controller-manager-645f4f9567-48w42\" (UID: \"5fffbf01-d444-4125-aef6-5fdaabad49cc\") " pod="kserve/kserve-controller-manager-645f4f9567-48w42"
Jul 20 10:06:00.423515 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.423401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76hv9\" (UniqueName: \"kubernetes.io/projected/5fffbf01-d444-4125-aef6-5fdaabad49cc-kube-api-access-76hv9\") pod \"kserve-controller-manager-645f4f9567-48w42\" (UID: \"5fffbf01-d444-4125-aef6-5fdaabad49cc\") " pod="kserve/kserve-controller-manager-645f4f9567-48w42"
Jul 20 10:06:00.423515 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.423423    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d1890299-2569-489f-9d25-a4a1b6786292-data\") pod \"seaweedfs-64568bcd49-pdprk\" (UID: \"d1890299-2569-489f-9d25-a4a1b6786292\") " pod="kserve/seaweedfs-64568bcd49-pdprk"
Jul 20 10:06:00.423515 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.423442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpzmg\" (UniqueName: \"kubernetes.io/projected/d1890299-2569-489f-9d25-a4a1b6786292-kube-api-access-tpzmg\") pod \"seaweedfs-64568bcd49-pdprk\" (UID: \"d1890299-2569-489f-9d25-a4a1b6786292\") " pod="kserve/seaweedfs-64568bcd49-pdprk"
Jul 20 10:06:00.425770 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.425746    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fffbf01-d444-4125-aef6-5fdaabad49cc-cert\") pod \"kserve-controller-manager-645f4f9567-48w42\" (UID: \"5fffbf01-d444-4125-aef6-5fdaabad49cc\") " pod="kserve/kserve-controller-manager-645f4f9567-48w42"
Jul 20 10:06:00.439593 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.439575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76hv9\" (UniqueName: \"kubernetes.io/projected/5fffbf01-d444-4125-aef6-5fdaabad49cc-kube-api-access-76hv9\") pod \"kserve-controller-manager-645f4f9567-48w42\" (UID: \"5fffbf01-d444-4125-aef6-5fdaabad49cc\") " pod="kserve/kserve-controller-manager-645f4f9567-48w42"
Jul 20 10:06:00.518633 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.518575    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-48w42"
Jul 20 10:06:00.524412 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.524395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d1890299-2569-489f-9d25-a4a1b6786292-data\") pod \"seaweedfs-64568bcd49-pdprk\" (UID: \"d1890299-2569-489f-9d25-a4a1b6786292\") " pod="kserve/seaweedfs-64568bcd49-pdprk"
Jul 20 10:06:00.524471 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.524423    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzmg\" (UniqueName: \"kubernetes.io/projected/d1890299-2569-489f-9d25-a4a1b6786292-kube-api-access-tpzmg\") pod \"seaweedfs-64568bcd49-pdprk\" (UID: \"d1890299-2569-489f-9d25-a4a1b6786292\") " pod="kserve/seaweedfs-64568bcd49-pdprk"
Jul 20 10:06:00.524751 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.524731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d1890299-2569-489f-9d25-a4a1b6786292-data\") pod \"seaweedfs-64568bcd49-pdprk\" (UID: \"d1890299-2569-489f-9d25-a4a1b6786292\") " pod="kserve/seaweedfs-64568bcd49-pdprk"
Jul 20 10:06:00.540205 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.540184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpzmg\" (UniqueName: \"kubernetes.io/projected/d1890299-2569-489f-9d25-a4a1b6786292-kube-api-access-tpzmg\") pod \"seaweedfs-64568bcd49-pdprk\" (UID: \"d1890299-2569-489f-9d25-a4a1b6786292\") " pod="kserve/seaweedfs-64568bcd49-pdprk"
Jul 20 10:06:00.583841 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.583817    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9hg6l\""
Jul 20 10:06:00.587939 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.587918    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-pdprk"
Jul 20 10:06:00.661667 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.661640    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-48w42"]
Jul 20 10:06:00.666470 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:06:00.666442    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fffbf01_d444_4125_aef6_5fdaabad49cc.slice/crio-f82a17108d5d4487f632ceca4fd188768d544c1575a517d0a7e07f7032ae9d1f WatchSource:0}: Error finding container f82a17108d5d4487f632ceca4fd188768d544c1575a517d0a7e07f7032ae9d1f: Status 404 returned error can't find the container with id f82a17108d5d4487f632ceca4fd188768d544c1575a517d0a7e07f7032ae9d1f
Jul 20 10:06:00.667848 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.667829    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:06:00.737386 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:00.737356    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-pdprk"]
Jul 20 10:06:00.742068 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:06:00.742045    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1890299_2569_489f_9d25_a4a1b6786292.slice/crio-4a139361925ccbc30ce2f86a91fb101f92b34a4b045aff47db52e4a208f4f0c1 WatchSource:0}: Error finding container 4a139361925ccbc30ce2f86a91fb101f92b34a4b045aff47db52e4a208f4f0c1: Status 404 returned error can't find the container with id 4a139361925ccbc30ce2f86a91fb101f92b34a4b045aff47db52e4a208f4f0c1
Jul 20 10:06:01.539503 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:01.539468    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-pdprk" event={"ID":"d1890299-2569-489f-9d25-a4a1b6786292","Type":"ContainerStarted","Data":"4a139361925ccbc30ce2f86a91fb101f92b34a4b045aff47db52e4a208f4f0c1"}
Jul 20 10:06:01.542099 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:01.542067    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-48w42" event={"ID":"5fffbf01-d444-4125-aef6-5fdaabad49cc","Type":"ContainerStarted","Data":"f82a17108d5d4487f632ceca4fd188768d544c1575a517d0a7e07f7032ae9d1f"}
Jul 20 10:06:04.314539 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:04.314450    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 20 10:06:04.555133 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:04.555085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-pdprk" event={"ID":"d1890299-2569-489f-9d25-a4a1b6786292","Type":"ContainerStarted","Data":"5af857b6dc760176211edf6e3724426a8c2218182239116a5b39c1ab01db97d1"}
Jul 20 10:06:04.555295 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:04.555182    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-pdprk"
Jul 20 10:06:04.556640 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:04.556587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-48w42" event={"ID":"5fffbf01-d444-4125-aef6-5fdaabad49cc","Type":"ContainerStarted","Data":"40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796"}
Jul 20 10:06:04.556721 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:04.556643    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-645f4f9567-48w42"
Jul 20 10:06:04.588396 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:04.588355    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-pdprk" podStartSLOduration=1.019903787 podStartE2EDuration="4.588341945s" totalImagesPullingTime="3.568438158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:06:04.586350868 +0000 UTC m=+544.424449472" watchObservedRunningTime="2026-07-20 10:06:04.588341945 +0000 UTC m=+544.426440530"
Jul 20 10:06:10.561834 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:10.561804    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-pdprk"
Jul 20 10:06:10.582865 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:10.582821    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-645f4f9567-48w42" podStartSLOduration=7.021752001 podStartE2EDuration="10.582804897s" totalImagesPullingTime="3.561052896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:06:04.650449565 +0000 UTC m=+544.488548148" watchObservedRunningTime="2026-07-20 10:06:10.582804897 +0000 UTC m=+550.420903476"
Jul 20 10:06:35.564888 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:35.564858    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-645f4f9567-48w42"
Jul 20 10:06:43.642869 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.642832    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-48w42"]
Jul 20 10:06:43.643495 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.643080    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-645f4f9567-48w42" podUID="5fffbf01-d444-4125-aef6-5fdaabad49cc" containerName="manager" containerID="cri-o://40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796" gracePeriod=10
Jul 20 10:06:43.683127 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.683091    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-w4qrj"]
Jul 20 10:06:43.685016 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.685002    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-w4qrj"
Jul 20 10:06:43.700065 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.700027    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-w4qrj"]
Jul 20 10:06:43.808588 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.808561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64r9c\" (UniqueName: \"kubernetes.io/projected/5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263-kube-api-access-64r9c\") pod \"kserve-controller-manager-645f4f9567-w4qrj\" (UID: \"5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263\") " pod="kserve/kserve-controller-manager-645f4f9567-w4qrj"
Jul 20 10:06:43.808708 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.808603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263-cert\") pod \"kserve-controller-manager-645f4f9567-w4qrj\" (UID: \"5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263\") " pod="kserve/kserve-controller-manager-645f4f9567-w4qrj"
Jul 20 10:06:43.873482 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.873456    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-48w42"
Jul 20 10:06:43.909624 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.909573    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76hv9\" (UniqueName: \"kubernetes.io/projected/5fffbf01-d444-4125-aef6-5fdaabad49cc-kube-api-access-76hv9\") pod \"5fffbf01-d444-4125-aef6-5fdaabad49cc\" (UID: \"5fffbf01-d444-4125-aef6-5fdaabad49cc\") "
Jul 20 10:06:43.909731 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.909627    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fffbf01-d444-4125-aef6-5fdaabad49cc-cert\") pod \"5fffbf01-d444-4125-aef6-5fdaabad49cc\" (UID: \"5fffbf01-d444-4125-aef6-5fdaabad49cc\") "
Jul 20 10:06:43.909776 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.909742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r9c\" (UniqueName: \"kubernetes.io/projected/5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263-kube-api-access-64r9c\") pod \"kserve-controller-manager-645f4f9567-w4qrj\" (UID: \"5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263\") " pod="kserve/kserve-controller-manager-645f4f9567-w4qrj"
Jul 20 10:06:43.909828 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.909782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263-cert\") pod \"kserve-controller-manager-645f4f9567-w4qrj\" (UID: \"5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263\") " pod="kserve/kserve-controller-manager-645f4f9567-w4qrj"
Jul 20 10:06:43.911807 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.911780    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fffbf01-d444-4125-aef6-5fdaabad49cc-kube-api-access-76hv9" (OuterVolumeSpecName: "kube-api-access-76hv9") pod "5fffbf01-d444-4125-aef6-5fdaabad49cc" (UID: "5fffbf01-d444-4125-aef6-5fdaabad49cc"). InnerVolumeSpecName "kube-api-access-76hv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:06:43.911999 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.911983    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fffbf01-d444-4125-aef6-5fdaabad49cc-cert" (OuterVolumeSpecName: "cert") pod "5fffbf01-d444-4125-aef6-5fdaabad49cc" (UID: "5fffbf01-d444-4125-aef6-5fdaabad49cc"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:06:43.912490 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.912473    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263-cert\") pod \"kserve-controller-manager-645f4f9567-w4qrj\" (UID: \"5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263\") " pod="kserve/kserve-controller-manager-645f4f9567-w4qrj"
Jul 20 10:06:43.918089 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:43.918069    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64r9c\" (UniqueName: \"kubernetes.io/projected/5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263-kube-api-access-64r9c\") pod \"kserve-controller-manager-645f4f9567-w4qrj\" (UID: \"5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263\") " pod="kserve/kserve-controller-manager-645f4f9567-w4qrj"
Jul 20 10:06:44.010146 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.010099    2561 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fffbf01-d444-4125-aef6-5fdaabad49cc-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:06:44.010146 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.010143    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76hv9\" (UniqueName: \"kubernetes.io/projected/5fffbf01-d444-4125-aef6-5fdaabad49cc-kube-api-access-76hv9\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:06:44.024252 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.024235    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-w4qrj"
Jul 20 10:06:44.151042 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.150999    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-w4qrj"]
Jul 20 10:06:44.154541 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:06:44.154517    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdeee7a_715e_4ca0_b0a3_9a55a6cc8263.slice/crio-9d2110760ac8e7b7cc2ccc9c970071229bcb879cc59f621b7f30821a20d36bf9 WatchSource:0}: Error finding container 9d2110760ac8e7b7cc2ccc9c970071229bcb879cc59f621b7f30821a20d36bf9: Status 404 returned error can't find the container with id 9d2110760ac8e7b7cc2ccc9c970071229bcb879cc59f621b7f30821a20d36bf9
Jul 20 10:06:44.683314 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.683279    2561 generic.go:356] "Generic (PLEG): container finished" podID="5fffbf01-d444-4125-aef6-5fdaabad49cc" containerID="40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796" exitCode=0
Jul 20 10:06:44.683759 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.683324    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-48w42"
Jul 20 10:06:44.683759 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.683363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-48w42" event={"ID":"5fffbf01-d444-4125-aef6-5fdaabad49cc","Type":"ContainerDied","Data":"40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796"}
Jul 20 10:06:44.683759 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.683418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-48w42" event={"ID":"5fffbf01-d444-4125-aef6-5fdaabad49cc","Type":"ContainerDied","Data":"f82a17108d5d4487f632ceca4fd188768d544c1575a517d0a7e07f7032ae9d1f"}
Jul 20 10:06:44.683759 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.683447    2561 scope.go:117] "RemoveContainer" containerID="40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796"
Jul 20 10:06:44.685020 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.684907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-w4qrj" event={"ID":"5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263","Type":"ContainerStarted","Data":"7d028ed9d09467d48025dc8daf451b8063ca6652c9ca46c878d7a057187c5b84"}
Jul 20 10:06:44.685020 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.684945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-w4qrj" event={"ID":"5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263","Type":"ContainerStarted","Data":"9d2110760ac8e7b7cc2ccc9c970071229bcb879cc59f621b7f30821a20d36bf9"}
Jul 20 10:06:44.685164 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.685059    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-645f4f9567-w4qrj"
Jul 20 10:06:44.692399 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.692384    2561 scope.go:117] "RemoveContainer" containerID="40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796"
Jul 20 10:06:44.692626 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:06:44.692610    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796\": container with ID starting with 40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796 not found: ID does not exist" containerID="40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796"
Jul 20 10:06:44.692678 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.692633    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796"} err="failed to get container status \"40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796\": rpc error: code = NotFound desc = could not find container \"40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796\": container with ID starting with 40c852cc66e895e8ce7b3d4b3ac1bbe5fe4917b1ff5729a85afd6f5262075796 not found: ID does not exist"
Jul 20 10:06:44.710800 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.710759    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-645f4f9567-w4qrj" podStartSLOduration=1.376201619 podStartE2EDuration="1.710748027s" totalImagesPullingTime="334.546408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:06:44.710528006 +0000 UTC m=+584.548626594" watchObservedRunningTime="2026-07-20 10:06:44.710748027 +0000 UTC m=+584.548846611"
Jul 20 10:06:44.726299 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.726278    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-48w42"]
Jul 20 10:06:44.735576 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.735557    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-48w42"]
Jul 20 10:06:44.764832 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:06:44.764780    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fffbf01-d444-4125-aef6-5fdaabad49cc" path="/var/lib/kubelet/pods/5fffbf01-d444-4125-aef6-5fdaabad49cc/volumes"
Jul 20 10:07:00.619826 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:00.619800    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:07:00.628031 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:00.628011    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:07:15.693415 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:15.693387    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-645f4f9567-w4qrj"
Jul 20 10:07:16.955233 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.955205    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-qjmq5"]
Jul 20 10:07:16.955690 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.955549    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fffbf01-d444-4125-aef6-5fdaabad49cc" containerName="manager"
Jul 20 10:07:16.955690 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.955566    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fffbf01-d444-4125-aef6-5fdaabad49cc" containerName="manager"
Jul 20 10:07:16.955690 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.955628    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fffbf01-d444-4125-aef6-5fdaabad49cc" containerName="manager"
Jul 20 10:07:16.957774 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.957751    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 10:07:16.958999 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.958977    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-zdxq7"]
Jul 20 10:07:16.960862 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.960841    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-zdxq7"
Jul 20 10:07:16.965458 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.965089    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 20 10:07:16.966259 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.966235    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-kx9pw\""
Jul 20 10:07:16.966259 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.966248    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 20 10:07:16.966259 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.966243    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-v8zcl\""
Jul 20 10:07:16.984072 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.984053    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-qjmq5"]
Jul 20 10:07:16.985341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:16.985321    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-zdxq7"]
Jul 20 10:07:17.024546 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.024523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f8e412-4a70-4059-9712-45d0ada332db-tls-certs\") pod \"model-serving-api-fd65d7d6-zdxq7\" (UID: \"38f8e412-4a70-4059-9712-45d0ada332db\") " pod="kserve/model-serving-api-fd65d7d6-zdxq7"
Jul 20 10:07:17.024645 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.024550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glcqx\" (UniqueName: \"kubernetes.io/projected/38f8e412-4a70-4059-9712-45d0ada332db-kube-api-access-glcqx\") pod \"model-serving-api-fd65d7d6-zdxq7\" (UID: \"38f8e412-4a70-4059-9712-45d0ada332db\") " pod="kserve/model-serving-api-fd65d7d6-zdxq7"
Jul 20 10:07:17.024645 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.024583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2665\" (UniqueName: \"kubernetes.io/projected/7d5831d2-0e10-454b-9fb5-fb909c11e317-kube-api-access-l2665\") pod \"odh-model-controller-84b54b7f97-qjmq5\" (UID: \"7d5831d2-0e10-454b-9fb5-fb909c11e317\") " pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 10:07:17.024645 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.024617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d5831d2-0e10-454b-9fb5-fb909c11e317-cert\") pod \"odh-model-controller-84b54b7f97-qjmq5\" (UID: \"7d5831d2-0e10-454b-9fb5-fb909c11e317\") " pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 10:07:17.125304 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.125277    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f8e412-4a70-4059-9712-45d0ada332db-tls-certs\") pod \"model-serving-api-fd65d7d6-zdxq7\" (UID: \"38f8e412-4a70-4059-9712-45d0ada332db\") " pod="kserve/model-serving-api-fd65d7d6-zdxq7"
Jul 20 10:07:17.125421 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.125311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glcqx\" (UniqueName: \"kubernetes.io/projected/38f8e412-4a70-4059-9712-45d0ada332db-kube-api-access-glcqx\") pod \"model-serving-api-fd65d7d6-zdxq7\" (UID: \"38f8e412-4a70-4059-9712-45d0ada332db\") " pod="kserve/model-serving-api-fd65d7d6-zdxq7"
Jul 20 10:07:17.125421 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.125344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2665\" (UniqueName: \"kubernetes.io/projected/7d5831d2-0e10-454b-9fb5-fb909c11e317-kube-api-access-l2665\") pod \"odh-model-controller-84b54b7f97-qjmq5\" (UID: \"7d5831d2-0e10-454b-9fb5-fb909c11e317\") " pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 10:07:17.125421 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.125362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d5831d2-0e10-454b-9fb5-fb909c11e317-cert\") pod \"odh-model-controller-84b54b7f97-qjmq5\" (UID: \"7d5831d2-0e10-454b-9fb5-fb909c11e317\") " pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 10:07:17.125590 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:07:17.125545    2561 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 20 10:07:17.125648 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:07:17.125600    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5831d2-0e10-454b-9fb5-fb909c11e317-cert podName:7d5831d2-0e10-454b-9fb5-fb909c11e317 nodeName:}" failed. No retries permitted until 2026-07-20 10:07:17.625582412 +0000 UTC m=+617.463680989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7d5831d2-0e10-454b-9fb5-fb909c11e317-cert") pod "odh-model-controller-84b54b7f97-qjmq5" (UID: "7d5831d2-0e10-454b-9fb5-fb909c11e317") : secret "odh-model-controller-webhook-cert" not found
Jul 20 10:07:17.127550 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.127534    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f8e412-4a70-4059-9712-45d0ada332db-tls-certs\") pod \"model-serving-api-fd65d7d6-zdxq7\" (UID: \"38f8e412-4a70-4059-9712-45d0ada332db\") " pod="kserve/model-serving-api-fd65d7d6-zdxq7"
Jul 20 10:07:17.155482 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.155459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glcqx\" (UniqueName: \"kubernetes.io/projected/38f8e412-4a70-4059-9712-45d0ada332db-kube-api-access-glcqx\") pod \"model-serving-api-fd65d7d6-zdxq7\" (UID: \"38f8e412-4a70-4059-9712-45d0ada332db\") " pod="kserve/model-serving-api-fd65d7d6-zdxq7"
Jul 20 10:07:17.164090 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.164070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2665\" (UniqueName: \"kubernetes.io/projected/7d5831d2-0e10-454b-9fb5-fb909c11e317-kube-api-access-l2665\") pod \"odh-model-controller-84b54b7f97-qjmq5\" (UID: \"7d5831d2-0e10-454b-9fb5-fb909c11e317\") " pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 10:07:17.274279 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.274216    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-zdxq7"
Jul 20 10:07:17.420831 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.420277    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-zdxq7"]
Jul 20 10:07:17.426504 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:07:17.426478    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f8e412_4a70_4059_9712_45d0ada332db.slice/crio-d2c59756cb1377997371a6746684415664b7602d1c1def1cee8df2bef64ca972 WatchSource:0}: Error finding container d2c59756cb1377997371a6746684415664b7602d1c1def1cee8df2bef64ca972: Status 404 returned error can't find the container with id d2c59756cb1377997371a6746684415664b7602d1c1def1cee8df2bef64ca972
Jul 20 10:07:17.628715 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.628681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d5831d2-0e10-454b-9fb5-fb909c11e317-cert\") pod \"odh-model-controller-84b54b7f97-qjmq5\" (UID: \"7d5831d2-0e10-454b-9fb5-fb909c11e317\") " pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 10:07:17.630956 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.630936    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d5831d2-0e10-454b-9fb5-fb909c11e317-cert\") pod \"odh-model-controller-84b54b7f97-qjmq5\" (UID: \"7d5831d2-0e10-454b-9fb5-fb909c11e317\") " pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 10:07:17.794970 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.794939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-zdxq7" event={"ID":"38f8e412-4a70-4059-9712-45d0ada332db","Type":"ContainerStarted","Data":"d2c59756cb1377997371a6746684415664b7602d1c1def1cee8df2bef64ca972"}
Jul 20 10:07:17.869333 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:17.869239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 10:07:18.028315 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:18.028282    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-qjmq5"]
Jul 20 10:07:18.031281 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:07:18.031228    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5831d2_0e10_454b_9fb5_fb909c11e317.slice/crio-4d5762ab68581d5509d4ec08132124aebd1d69ce558ea37d72ac8bf87952e7f6 WatchSource:0}: Error finding container 4d5762ab68581d5509d4ec08132124aebd1d69ce558ea37d72ac8bf87952e7f6: Status 404 returned error can't find the container with id 4d5762ab68581d5509d4ec08132124aebd1d69ce558ea37d72ac8bf87952e7f6
Jul 20 10:07:18.801369 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:18.801296    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-qjmq5" event={"ID":"7d5831d2-0e10-454b-9fb5-fb909c11e317","Type":"ContainerStarted","Data":"4d5762ab68581d5509d4ec08132124aebd1d69ce558ea37d72ac8bf87952e7f6"}
Jul 20 10:07:19.806739 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:19.806694    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-zdxq7" event={"ID":"38f8e412-4a70-4059-9712-45d0ada332db","Type":"ContainerStarted","Data":"389251a758c3a9b6c0c023e04c54ce0af685a4010ea5b566b0caa8041bbcdf73"}
Jul 20 10:07:19.807174 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:19.806790    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-zdxq7"
Jul 20 10:07:19.826745 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:19.826696    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-zdxq7" podStartSLOduration=2.12095645 podStartE2EDuration="3.826677415s" totalImagesPullingTime="1.705720965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:07:19.826251126 +0000 UTC m=+619.664349732" watchObservedRunningTime="2026-07-20 10:07:19.826677415 +0000 UTC m=+619.664776003"
Jul 20 10:07:20.812305 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:20.812208    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-qjmq5" event={"ID":"7d5831d2-0e10-454b-9fb5-fb909c11e317","Type":"ContainerStarted","Data":"93b6449580d39cb45b8ab33bf09e2232b77b8fa69ad36ffe64a81ea0075351a1"}
Jul 20 10:07:20.812305 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:20.812284    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 10:07:20.838642 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:20.838598    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-qjmq5" podStartSLOduration=2.432245456 podStartE2EDuration="4.838584603s" totalImagesPullingTime="2.406339147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:07:20.836798951 +0000 UTC m=+620.674897536" watchObservedRunningTime="2026-07-20 10:07:20.838584603 +0000 UTC m=+620.676683187"
Jul 20 10:07:30.816702 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:30.816627    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-zdxq7"
Jul 20 10:07:31.817347 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:31.817322    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 10:07:32.699963 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:32.699928    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-2cpq9"]
Jul 20 10:07:32.702947 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:32.702932    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2cpq9"
Jul 20 10:07:32.712635 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:32.712611    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2cpq9"]
Jul 20 10:07:32.833985 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:32.833958    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg4jf\" (UniqueName: \"kubernetes.io/projected/a1a02fda-409f-4d26-90d4-ba4a0eb041a2-kube-api-access-wg4jf\") pod \"s3-init-2cpq9\" (UID: \"a1a02fda-409f-4d26-90d4-ba4a0eb041a2\") " pod="kserve/s3-init-2cpq9"
Jul 20 10:07:32.834349 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:32.833997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a02fda-409f-4d26-90d4-ba4a0eb041a2-model-cache\") pod \"s3-init-2cpq9\" (UID: \"a1a02fda-409f-4d26-90d4-ba4a0eb041a2\") " pod="kserve/s3-init-2cpq9"
Jul 20 10:07:32.934549 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:32.934527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a02fda-409f-4d26-90d4-ba4a0eb041a2-model-cache\") pod \"s3-init-2cpq9\" (UID: \"a1a02fda-409f-4d26-90d4-ba4a0eb041a2\") " pod="kserve/s3-init-2cpq9"
Jul 20 10:07:32.934671 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:32.934579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg4jf\" (UniqueName: \"kubernetes.io/projected/a1a02fda-409f-4d26-90d4-ba4a0eb041a2-kube-api-access-wg4jf\") pod \"s3-init-2cpq9\" (UID: \"a1a02fda-409f-4d26-90d4-ba4a0eb041a2\") " pod="kserve/s3-init-2cpq9"
Jul 20 10:07:32.934863 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:32.934846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a02fda-409f-4d26-90d4-ba4a0eb041a2-model-cache\") pod \"s3-init-2cpq9\" (UID: \"a1a02fda-409f-4d26-90d4-ba4a0eb041a2\") " pod="kserve/s3-init-2cpq9"
Jul 20 10:07:32.943486 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:32.943463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg4jf\" (UniqueName: \"kubernetes.io/projected/a1a02fda-409f-4d26-90d4-ba4a0eb041a2-kube-api-access-wg4jf\") pod \"s3-init-2cpq9\" (UID: \"a1a02fda-409f-4d26-90d4-ba4a0eb041a2\") " pod="kserve/s3-init-2cpq9"
Jul 20 10:07:33.012043 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:33.011993    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2cpq9"
Jul 20 10:07:33.166796 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:33.166683    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2cpq9"]
Jul 20 10:07:33.169262 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:07:33.169235    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1a02fda_409f_4d26_90d4_ba4a0eb041a2.slice/crio-5c6eaf38ae0eee0a929bcddb4297b655354410d2977dd765dfaf977fdceab45f WatchSource:0}: Error finding container 5c6eaf38ae0eee0a929bcddb4297b655354410d2977dd765dfaf977fdceab45f: Status 404 returned error can't find the container with id 5c6eaf38ae0eee0a929bcddb4297b655354410d2977dd765dfaf977fdceab45f
Jul 20 10:07:33.853010 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:33.852972    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2cpq9" event={"ID":"a1a02fda-409f-4d26-90d4-ba4a0eb041a2","Type":"ContainerStarted","Data":"5c6eaf38ae0eee0a929bcddb4297b655354410d2977dd765dfaf977fdceab45f"}
Jul 20 10:07:35.861637 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:35.861611    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2cpq9" event={"ID":"a1a02fda-409f-4d26-90d4-ba4a0eb041a2","Type":"ContainerStarted","Data":"cd6104327cc6e6a22c427e2b512b88179f5df717858c3b80e6192ec507e71f84"}
Jul 20 10:07:37.876892 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:37.876858    2561 generic.go:356] "Generic (PLEG): container finished" podID="a1a02fda-409f-4d26-90d4-ba4a0eb041a2" containerID="cd6104327cc6e6a22c427e2b512b88179f5df717858c3b80e6192ec507e71f84" exitCode=0
Jul 20 10:07:37.877266 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:37.876933    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2cpq9" event={"ID":"a1a02fda-409f-4d26-90d4-ba4a0eb041a2","Type":"ContainerDied","Data":"cd6104327cc6e6a22c427e2b512b88179f5df717858c3b80e6192ec507e71f84"}
Jul 20 10:07:42.898379 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:42.898347    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2cpq9" event={"ID":"a1a02fda-409f-4d26-90d4-ba4a0eb041a2","Type":"ContainerStarted","Data":"c706b3ed825ac8b5e9b6dff5a9b76347acb65fe7f66c4215ce95bd79f635c9ff"}
Jul 20 10:07:42.922448 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:07:42.922389    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-2cpq9" podStartSLOduration=2.881660115 podStartE2EDuration="10.922367657s" totalImagesPullingTime="6.931021257s" totalInitContainerRuntime="1.109686285s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:07:42.91988339 +0000 UTC m=+642.757981975" watchObservedRunningTime="2026-07-20 10:07:42.922367657 +0000 UTC m=+642.760466243"
Jul 20 10:08:48.121948 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:48.121917    2561 generic.go:356] "Generic (PLEG): container finished" podID="a1a02fda-409f-4d26-90d4-ba4a0eb041a2" containerID="c706b3ed825ac8b5e9b6dff5a9b76347acb65fe7f66c4215ce95bd79f635c9ff" exitCode=0
Jul 20 10:08:48.122366 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:48.121992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2cpq9" event={"ID":"a1a02fda-409f-4d26-90d4-ba4a0eb041a2","Type":"ContainerDied","Data":"c706b3ed825ac8b5e9b6dff5a9b76347acb65fe7f66c4215ce95bd79f635c9ff"}
Jul 20 10:08:49.249158 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:49.249136    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2cpq9"
Jul 20 10:08:49.352052 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:49.352021    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg4jf\" (UniqueName: \"kubernetes.io/projected/a1a02fda-409f-4d26-90d4-ba4a0eb041a2-kube-api-access-wg4jf\") pod \"a1a02fda-409f-4d26-90d4-ba4a0eb041a2\" (UID: \"a1a02fda-409f-4d26-90d4-ba4a0eb041a2\") "
Jul 20 10:08:49.352208 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:49.352059    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a02fda-409f-4d26-90d4-ba4a0eb041a2-model-cache\") pod \"a1a02fda-409f-4d26-90d4-ba4a0eb041a2\" (UID: \"a1a02fda-409f-4d26-90d4-ba4a0eb041a2\") "
Jul 20 10:08:49.354017 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:49.353988    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a02fda-409f-4d26-90d4-ba4a0eb041a2-kube-api-access-wg4jf" (OuterVolumeSpecName: "kube-api-access-wg4jf") pod "a1a02fda-409f-4d26-90d4-ba4a0eb041a2" (UID: "a1a02fda-409f-4d26-90d4-ba4a0eb041a2"). InnerVolumeSpecName "kube-api-access-wg4jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:08:49.370466 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:49.370438    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a02fda-409f-4d26-90d4-ba4a0eb041a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1a02fda-409f-4d26-90d4-ba4a0eb041a2" (UID: "a1a02fda-409f-4d26-90d4-ba4a0eb041a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:08:49.452595 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:49.452538    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg4jf\" (UniqueName: \"kubernetes.io/projected/a1a02fda-409f-4d26-90d4-ba4a0eb041a2-kube-api-access-wg4jf\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:08:49.452595 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:49.452562    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a02fda-409f-4d26-90d4-ba4a0eb041a2-model-cache\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:08:50.129845 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:50.129808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2cpq9" event={"ID":"a1a02fda-409f-4d26-90d4-ba4a0eb041a2","Type":"ContainerDied","Data":"5c6eaf38ae0eee0a929bcddb4297b655354410d2977dd765dfaf977fdceab45f"}
Jul 20 10:08:50.129845 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:50.129846    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c6eaf38ae0eee0a929bcddb4297b655354410d2977dd765dfaf977fdceab45f"
Jul 20 10:08:50.129845 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:50.129824    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2cpq9"
Jul 20 10:08:51.048297 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.048269    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"]
Jul 20 10:08:51.048682 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.048497    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a02fda-409f-4d26-90d4-ba4a0eb041a2" containerName="s3-init"
Jul 20 10:08:51.048682 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.048507    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a02fda-409f-4d26-90d4-ba4a0eb041a2" containerName="s3-init"
Jul 20 10:08:51.048682 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.048527    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a02fda-409f-4d26-90d4-ba4a0eb041a2" containerName="download-hf-model"
Jul 20 10:08:51.048682 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.048533    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a02fda-409f-4d26-90d4-ba4a0eb041a2" containerName="download-hf-model"
Jul 20 10:08:51.048682 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.048573    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a02fda-409f-4d26-90d4-ba4a0eb041a2" containerName="s3-init"
Jul 20 10:08:51.051442 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.051427    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"
Jul 20 10:08:51.054108 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.054088    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 20 10:08:51.062620 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.062600    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"]
Jul 20 10:08:51.165332 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.165307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/66d60509-a534-448c-b365-0303de844a58-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-d4f5t\" (UID: \"66d60509-a534-448c-b365-0303de844a58\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"
Jul 20 10:08:51.165459 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.165338    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qth7v\" (UniqueName: \"kubernetes.io/projected/66d60509-a534-448c-b365-0303de844a58-kube-api-access-qth7v\") pod \"seaweedfs-tls-custom-6644bf5d6c-d4f5t\" (UID: \"66d60509-a534-448c-b365-0303de844a58\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"
Jul 20 10:08:51.266282 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.266252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/66d60509-a534-448c-b365-0303de844a58-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-d4f5t\" (UID: \"66d60509-a534-448c-b365-0303de844a58\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"
Jul 20 10:08:51.266388 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.266292    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qth7v\" (UniqueName: \"kubernetes.io/projected/66d60509-a534-448c-b365-0303de844a58-kube-api-access-qth7v\") pod \"seaweedfs-tls-custom-6644bf5d6c-d4f5t\" (UID: \"66d60509-a534-448c-b365-0303de844a58\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"
Jul 20 10:08:51.266614 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.266596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/66d60509-a534-448c-b365-0303de844a58-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-d4f5t\" (UID: \"66d60509-a534-448c-b365-0303de844a58\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"
Jul 20 10:08:51.276022 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.275995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qth7v\" (UniqueName: \"kubernetes.io/projected/66d60509-a534-448c-b365-0303de844a58-kube-api-access-qth7v\") pod \"seaweedfs-tls-custom-6644bf5d6c-d4f5t\" (UID: \"66d60509-a534-448c-b365-0303de844a58\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"
Jul 20 10:08:51.359554 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.359497    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"
Jul 20 10:08:51.485552 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:51.485520    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"]
Jul 20 10:08:51.488380 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:08:51.488353    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d60509_a534_448c_b365_0303de844a58.slice/crio-46ffab2ddbbf52b8acfa8bd8c9f912708c36f29d9eef6f3ebdd942c3a92ca7ec WatchSource:0}: Error finding container 46ffab2ddbbf52b8acfa8bd8c9f912708c36f29d9eef6f3ebdd942c3a92ca7ec: Status 404 returned error can't find the container with id 46ffab2ddbbf52b8acfa8bd8c9f912708c36f29d9eef6f3ebdd942c3a92ca7ec
Jul 20 10:08:52.138037 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:52.138003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t" event={"ID":"66d60509-a534-448c-b365-0303de844a58","Type":"ContainerStarted","Data":"9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124"}
Jul 20 10:08:52.138424 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:52.138059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t" event={"ID":"66d60509-a534-448c-b365-0303de844a58","Type":"ContainerStarted","Data":"46ffab2ddbbf52b8acfa8bd8c9f912708c36f29d9eef6f3ebdd942c3a92ca7ec"}
Jul 20 10:08:52.155743 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:52.155703    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t" podStartSLOduration=0.870726961 podStartE2EDuration="1.155685172s" totalImagesPullingTime="284.958211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:08:52.154755659 +0000 UTC m=+711.992854244" watchObservedRunningTime="2026-07-20 10:08:52.155685172 +0000 UTC m=+711.993783757"
Jul 20 10:08:53.479285 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:53.479205    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"]
Jul 20 10:08:54.145595 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:54.145556    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t" podUID="66d60509-a534-448c-b365-0303de844a58" containerName="seaweedfs-tls-custom" containerID="cri-o://9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124" gracePeriod=30
Jul 20 10:08:55.381604 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:55.381581    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"
Jul 20 10:08:55.496563 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:55.496486    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/66d60509-a534-448c-b365-0303de844a58-data\") pod \"66d60509-a534-448c-b365-0303de844a58\" (UID: \"66d60509-a534-448c-b365-0303de844a58\") "
Jul 20 10:08:55.496563 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:55.496515    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qth7v\" (UniqueName: \"kubernetes.io/projected/66d60509-a534-448c-b365-0303de844a58-kube-api-access-qth7v\") pod \"66d60509-a534-448c-b365-0303de844a58\" (UID: \"66d60509-a534-448c-b365-0303de844a58\") "
Jul 20 10:08:55.497644 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:55.497622    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d60509-a534-448c-b365-0303de844a58-data" (OuterVolumeSpecName: "data") pod "66d60509-a534-448c-b365-0303de844a58" (UID: "66d60509-a534-448c-b365-0303de844a58"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:08:55.498499 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:55.498478    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d60509-a534-448c-b365-0303de844a58-kube-api-access-qth7v" (OuterVolumeSpecName: "kube-api-access-qth7v") pod "66d60509-a534-448c-b365-0303de844a58" (UID: "66d60509-a534-448c-b365-0303de844a58"). InnerVolumeSpecName "kube-api-access-qth7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:08:55.597679 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:55.597655    2561 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/66d60509-a534-448c-b365-0303de844a58-data\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:08:55.597679 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:55.597679    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qth7v\" (UniqueName: \"kubernetes.io/projected/66d60509-a534-448c-b365-0303de844a58-kube-api-access-qth7v\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:08:56.152651 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:56.152615    2561 generic.go:356] "Generic (PLEG): container finished" podID="66d60509-a534-448c-b365-0303de844a58" containerID="9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124" exitCode=0
Jul 20 10:08:56.152651 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:56.152654    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"
Jul 20 10:08:56.152901 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:56.152695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t" event={"ID":"66d60509-a534-448c-b365-0303de844a58","Type":"ContainerDied","Data":"9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124"}
Jul 20 10:08:56.152901 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:56.152745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t" event={"ID":"66d60509-a534-448c-b365-0303de844a58","Type":"ContainerDied","Data":"46ffab2ddbbf52b8acfa8bd8c9f912708c36f29d9eef6f3ebdd942c3a92ca7ec"}
Jul 20 10:08:56.152901 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:56.152766    2561 scope.go:117] "RemoveContainer" containerID="9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124"
Jul 20 10:08:56.161748 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:56.161731    2561 scope.go:117] "RemoveContainer" containerID="9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124"
Jul 20 10:08:56.161984 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:08:56.161964    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124\": container with ID starting with 9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124 not found: ID does not exist" containerID="9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124"
Jul 20 10:08:56.162045 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:56.161996    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124"} err="failed to get container status \"9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124\": rpc error: code = NotFound desc = could not find container \"9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124\": container with ID starting with 9bb28dab341e05f0a502e684850b8ed9b57769768d27238b096c389a8090d124 not found: ID does not exist"
Jul 20 10:08:56.190996 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:56.190975    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"]
Jul 20 10:08:56.199572 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:56.199549    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-d4f5t"]
Jul 20 10:08:56.765091 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:08:56.765052    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d60509-a534-448c-b365-0303de844a58" path="/var/lib/kubelet/pods/66d60509-a534-448c-b365-0303de844a58/volumes"
Jul 20 10:09:01.345377 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.345346    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-6qztd"]
Jul 20 10:09:01.345738 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.345623    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d60509-a534-448c-b365-0303de844a58" containerName="seaweedfs-tls-custom"
Jul 20 10:09:01.345738 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.345634    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d60509-a534-448c-b365-0303de844a58" containerName="seaweedfs-tls-custom"
Jul 20 10:09:01.345738 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.345677    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d60509-a534-448c-b365-0303de844a58" containerName="seaweedfs-tls-custom"
Jul 20 10:09:01.349950 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.349933    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6qztd"
Jul 20 10:09:01.353365 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.353342    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 20 10:09:01.363972 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.363941    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6qztd"]
Jul 20 10:09:01.438466 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.438440    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p64d\" (UniqueName: \"kubernetes.io/projected/c811e518-aeb6-4346-bfd7-4b68ce20ba48-kube-api-access-2p64d\") pod \"s3-tls-init-custom-6qztd\" (UID: \"c811e518-aeb6-4346-bfd7-4b68ce20ba48\") " pod="kserve/s3-tls-init-custom-6qztd"
Jul 20 10:09:01.438572 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.438480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c811e518-aeb6-4346-bfd7-4b68ce20ba48-model-cache\") pod \"s3-tls-init-custom-6qztd\" (UID: \"c811e518-aeb6-4346-bfd7-4b68ce20ba48\") " pod="kserve/s3-tls-init-custom-6qztd"
Jul 20 10:09:01.538755 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.538728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p64d\" (UniqueName: \"kubernetes.io/projected/c811e518-aeb6-4346-bfd7-4b68ce20ba48-kube-api-access-2p64d\") pod \"s3-tls-init-custom-6qztd\" (UID: \"c811e518-aeb6-4346-bfd7-4b68ce20ba48\") " pod="kserve/s3-tls-init-custom-6qztd"
Jul 20 10:09:01.538861 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.538766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c811e518-aeb6-4346-bfd7-4b68ce20ba48-model-cache\") pod \"s3-tls-init-custom-6qztd\" (UID: \"c811e518-aeb6-4346-bfd7-4b68ce20ba48\") " pod="kserve/s3-tls-init-custom-6qztd"
Jul 20 10:09:01.539092 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.539076    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c811e518-aeb6-4346-bfd7-4b68ce20ba48-model-cache\") pod \"s3-tls-init-custom-6qztd\" (UID: \"c811e518-aeb6-4346-bfd7-4b68ce20ba48\") " pod="kserve/s3-tls-init-custom-6qztd"
Jul 20 10:09:01.551506 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.551480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p64d\" (UniqueName: \"kubernetes.io/projected/c811e518-aeb6-4346-bfd7-4b68ce20ba48-kube-api-access-2p64d\") pod \"s3-tls-init-custom-6qztd\" (UID: \"c811e518-aeb6-4346-bfd7-4b68ce20ba48\") " pod="kserve/s3-tls-init-custom-6qztd"
Jul 20 10:09:01.658692 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.658662    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6qztd"
Jul 20 10:09:01.792942 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:01.792914    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6qztd"]
Jul 20 10:09:01.797417 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:09:01.797390    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc811e518_aeb6_4346_bfd7_4b68ce20ba48.slice/crio-769d0b0581dcb086743f4184f04c5b746d10a8ffe27b898ef39a71ddf9105258 WatchSource:0}: Error finding container 769d0b0581dcb086743f4184f04c5b746d10a8ffe27b898ef39a71ddf9105258: Status 404 returned error can't find the container with id 769d0b0581dcb086743f4184f04c5b746d10a8ffe27b898ef39a71ddf9105258
Jul 20 10:09:02.180489 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:02.180448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6qztd" event={"ID":"c811e518-aeb6-4346-bfd7-4b68ce20ba48","Type":"ContainerStarted","Data":"6195d6163eb49f734b3ddf36a59c7bf2f99ee2de501166861b75671dcf44dfec"}
Jul 20 10:09:02.180646 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:02.180505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6qztd" event={"ID":"c811e518-aeb6-4346-bfd7-4b68ce20ba48","Type":"ContainerStarted","Data":"769d0b0581dcb086743f4184f04c5b746d10a8ffe27b898ef39a71ddf9105258"}
Jul 20 10:09:03.185783 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:03.185755    2561 generic.go:356] "Generic (PLEG): container finished" podID="c811e518-aeb6-4346-bfd7-4b68ce20ba48" containerID="6195d6163eb49f734b3ddf36a59c7bf2f99ee2de501166861b75671dcf44dfec" exitCode=0
Jul 20 10:09:03.186126 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:03.185809    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6qztd" event={"ID":"c811e518-aeb6-4346-bfd7-4b68ce20ba48","Type":"ContainerDied","Data":"6195d6163eb49f734b3ddf36a59c7bf2f99ee2de501166861b75671dcf44dfec"}
Jul 20 10:09:04.190425 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:04.190389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6qztd" event={"ID":"c811e518-aeb6-4346-bfd7-4b68ce20ba48","Type":"ContainerStarted","Data":"07fa51ff959c10b3aa830fa29cf561786a652e6f1fd38f98d2f62a2a255a8a51"}
Jul 20 10:09:04.208885 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:04.208832    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-6qztd" podStartSLOduration=1.914299489 podStartE2EDuration="3.208813569s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.29451408s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:09:04.207739015 +0000 UTC m=+724.045837600" watchObservedRunningTime="2026-07-20 10:09:04.208813569 +0000 UTC m=+724.046912155"
Jul 20 10:09:10.214913 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:10.214880    2561 generic.go:356] "Generic (PLEG): container finished" podID="c811e518-aeb6-4346-bfd7-4b68ce20ba48" containerID="07fa51ff959c10b3aa830fa29cf561786a652e6f1fd38f98d2f62a2a255a8a51" exitCode=0
Jul 20 10:09:10.215388 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:10.214919    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6qztd" event={"ID":"c811e518-aeb6-4346-bfd7-4b68ce20ba48","Type":"ContainerDied","Data":"07fa51ff959c10b3aa830fa29cf561786a652e6f1fd38f98d2f62a2a255a8a51"}
Jul 20 10:09:11.353173 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:11.353147    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6qztd"
Jul 20 10:09:11.507605 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:11.507529    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c811e518-aeb6-4346-bfd7-4b68ce20ba48-model-cache\") pod \"c811e518-aeb6-4346-bfd7-4b68ce20ba48\" (UID: \"c811e518-aeb6-4346-bfd7-4b68ce20ba48\") "
Jul 20 10:09:11.507605 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:11.507605    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p64d\" (UniqueName: \"kubernetes.io/projected/c811e518-aeb6-4346-bfd7-4b68ce20ba48-kube-api-access-2p64d\") pod \"c811e518-aeb6-4346-bfd7-4b68ce20ba48\" (UID: \"c811e518-aeb6-4346-bfd7-4b68ce20ba48\") "
Jul 20 10:09:11.509569 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:11.509547    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c811e518-aeb6-4346-bfd7-4b68ce20ba48-kube-api-access-2p64d" (OuterVolumeSpecName: "kube-api-access-2p64d") pod "c811e518-aeb6-4346-bfd7-4b68ce20ba48" (UID: "c811e518-aeb6-4346-bfd7-4b68ce20ba48"). InnerVolumeSpecName "kube-api-access-2p64d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:09:11.528635 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:11.528608    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c811e518-aeb6-4346-bfd7-4b68ce20ba48-model-cache" (OuterVolumeSpecName: "model-cache") pod "c811e518-aeb6-4346-bfd7-4b68ce20ba48" (UID: "c811e518-aeb6-4346-bfd7-4b68ce20ba48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:09:11.608173 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:11.608152    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p64d\" (UniqueName: \"kubernetes.io/projected/c811e518-aeb6-4346-bfd7-4b68ce20ba48-kube-api-access-2p64d\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:09:11.608173 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:11.608171    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c811e518-aeb6-4346-bfd7-4b68ce20ba48-model-cache\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:09:12.223640 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.223599    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6qztd" event={"ID":"c811e518-aeb6-4346-bfd7-4b68ce20ba48","Type":"ContainerDied","Data":"769d0b0581dcb086743f4184f04c5b746d10a8ffe27b898ef39a71ddf9105258"}
Jul 20 10:09:12.223640 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.223620    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6qztd"
Jul 20 10:09:12.223640 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.223639    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="769d0b0581dcb086743f4184f04c5b746d10a8ffe27b898ef39a71ddf9105258"
Jul 20 10:09:12.906834 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.906800    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"]
Jul 20 10:09:12.907226 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.907031    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c811e518-aeb6-4346-bfd7-4b68ce20ba48" containerName="download-hf-model"
Jul 20 10:09:12.907226 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.907041    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c811e518-aeb6-4346-bfd7-4b68ce20ba48" containerName="download-hf-model"
Jul 20 10:09:12.907226 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.907054    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c811e518-aeb6-4346-bfd7-4b68ce20ba48" containerName="s3-tls-init-custom"
Jul 20 10:09:12.907226 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.907059    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c811e518-aeb6-4346-bfd7-4b68ce20ba48" containerName="s3-tls-init-custom"
Jul 20 10:09:12.907226 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.907104    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c811e518-aeb6-4346-bfd7-4b68ce20ba48" containerName="s3-tls-init-custom"
Jul 20 10:09:12.910836 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.910820    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"
Jul 20 10:09:12.913197 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.913173    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 20 10:09:12.913577 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.913563    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 20 10:09:12.918604 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:12.918582    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"]
Jul 20 10:09:13.016590 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:13.016562    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6823c38-491b-43b7-9bde-50ea7cec69f8-data\") pod \"seaweedfs-tls-serving-67b7477c6d-ggh99\" (UID: \"d6823c38-491b-43b7-9bde-50ea7cec69f8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"
Jul 20 10:09:13.016708 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:13.016596    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88wp\" (UniqueName: \"kubernetes.io/projected/d6823c38-491b-43b7-9bde-50ea7cec69f8-kube-api-access-d88wp\") pod \"seaweedfs-tls-serving-67b7477c6d-ggh99\" (UID: \"d6823c38-491b-43b7-9bde-50ea7cec69f8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"
Jul 20 10:09:13.016708 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:13.016675    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/d6823c38-491b-43b7-9bde-50ea7cec69f8-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-ggh99\" (UID: \"d6823c38-491b-43b7-9bde-50ea7cec69f8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"
Jul 20 10:09:13.117685 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:13.117660    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6823c38-491b-43b7-9bde-50ea7cec69f8-data\") pod \"seaweedfs-tls-serving-67b7477c6d-ggh99\" (UID: \"d6823c38-491b-43b7-9bde-50ea7cec69f8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"
Jul 20 10:09:13.117801 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:13.117690    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88wp\" (UniqueName: \"kubernetes.io/projected/d6823c38-491b-43b7-9bde-50ea7cec69f8-kube-api-access-d88wp\") pod \"seaweedfs-tls-serving-67b7477c6d-ggh99\" (UID: \"d6823c38-491b-43b7-9bde-50ea7cec69f8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"
Jul 20 10:09:13.117801 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:13.117729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/d6823c38-491b-43b7-9bde-50ea7cec69f8-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-ggh99\" (UID: \"d6823c38-491b-43b7-9bde-50ea7cec69f8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"
Jul 20 10:09:13.117867 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:09:13.117834    2561 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 20 10:09:13.117867 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:09:13.117846    2561 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-ggh99: secret "seaweedfs-tls-serving" not found
Jul 20 10:09:13.117929 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:09:13.117901    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6823c38-491b-43b7-9bde-50ea7cec69f8-seaweedfs-tls-serving podName:d6823c38-491b-43b7-9bde-50ea7cec69f8 nodeName:}" failed. No retries permitted until 2026-07-20 10:09:13.617884522 +0000 UTC m=+733.455983103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/d6823c38-491b-43b7-9bde-50ea7cec69f8-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-ggh99" (UID: "d6823c38-491b-43b7-9bde-50ea7cec69f8") : secret "seaweedfs-tls-serving" not found
Jul 20 10:09:13.117999 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:13.117982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6823c38-491b-43b7-9bde-50ea7cec69f8-data\") pod \"seaweedfs-tls-serving-67b7477c6d-ggh99\" (UID: \"d6823c38-491b-43b7-9bde-50ea7cec69f8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"
Jul 20 10:09:13.126471 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:13.126445    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88wp\" (UniqueName: \"kubernetes.io/projected/d6823c38-491b-43b7-9bde-50ea7cec69f8-kube-api-access-d88wp\") pod \"seaweedfs-tls-serving-67b7477c6d-ggh99\" (UID: \"d6823c38-491b-43b7-9bde-50ea7cec69f8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"
Jul 20 10:09:13.621566 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:13.621527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/d6823c38-491b-43b7-9bde-50ea7cec69f8-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-ggh99\" (UID: \"d6823c38-491b-43b7-9bde-50ea7cec69f8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"
Jul 20 10:09:13.623875 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:13.623845    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/d6823c38-491b-43b7-9bde-50ea7cec69f8-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-ggh99\" (UID: \"d6823c38-491b-43b7-9bde-50ea7cec69f8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"
Jul 20 10:09:13.819495 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:13.819451    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"
Jul 20 10:09:13.969657 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:13.969510    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-ggh99"]
Jul 20 10:09:13.972208 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:09:13.972178    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6823c38_491b_43b7_9bde_50ea7cec69f8.slice/crio-86c930b2d651f71c4cd2b6aaab245349648d0eb7cd135f6429e51d83aae952ae WatchSource:0}: Error finding container 86c930b2d651f71c4cd2b6aaab245349648d0eb7cd135f6429e51d83aae952ae: Status 404 returned error can't find the container with id 86c930b2d651f71c4cd2b6aaab245349648d0eb7cd135f6429e51d83aae952ae
Jul 20 10:09:14.231170 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:14.231086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99" event={"ID":"d6823c38-491b-43b7-9bde-50ea7cec69f8","Type":"ContainerStarted","Data":"86c930b2d651f71c4cd2b6aaab245349648d0eb7cd135f6429e51d83aae952ae"}
Jul 20 10:09:15.235241 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:15.235209    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99" event={"ID":"d6823c38-491b-43b7-9bde-50ea7cec69f8","Type":"ContainerStarted","Data":"0fa33aba74f90dbf3120001a6770ebb4ff72685fd74d02e2425ada7037183d8a"}
Jul 20 10:09:15.254820 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:15.254770    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-ggh99" podStartSLOduration=2.952449985 podStartE2EDuration="3.254755568s" totalImagesPullingTime="302.305583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:09:15.253260364 +0000 UTC m=+735.091358949" watchObservedRunningTime="2026-07-20 10:09:15.254755568 +0000 UTC m=+735.092854272"
Jul 20 10:09:15.848072 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:15.848033    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-kttst"]
Jul 20 10:09:15.852479 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:15.852455    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-kttst"
Jul 20 10:09:15.859472 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:15.859447    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-kttst"]
Jul 20 10:09:15.936693 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:15.936668    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93836ca5-dbd2-421f-8c04-a14b44d0da8e-model-cache\") pod \"s3-tls-init-serving-kttst\" (UID: \"93836ca5-dbd2-421f-8c04-a14b44d0da8e\") " pod="kserve/s3-tls-init-serving-kttst"
Jul 20 10:09:15.936817 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:15.936708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2z96\" (UniqueName: \"kubernetes.io/projected/93836ca5-dbd2-421f-8c04-a14b44d0da8e-kube-api-access-w2z96\") pod \"s3-tls-init-serving-kttst\" (UID: \"93836ca5-dbd2-421f-8c04-a14b44d0da8e\") " pod="kserve/s3-tls-init-serving-kttst"
Jul 20 10:09:16.037239 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:16.037209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93836ca5-dbd2-421f-8c04-a14b44d0da8e-model-cache\") pod \"s3-tls-init-serving-kttst\" (UID: \"93836ca5-dbd2-421f-8c04-a14b44d0da8e\") " pod="kserve/s3-tls-init-serving-kttst"
Jul 20 10:09:16.037363 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:16.037256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2z96\" (UniqueName: \"kubernetes.io/projected/93836ca5-dbd2-421f-8c04-a14b44d0da8e-kube-api-access-w2z96\") pod \"s3-tls-init-serving-kttst\" (UID: \"93836ca5-dbd2-421f-8c04-a14b44d0da8e\") " pod="kserve/s3-tls-init-serving-kttst"
Jul 20 10:09:16.037569 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:16.037552    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93836ca5-dbd2-421f-8c04-a14b44d0da8e-model-cache\") pod \"s3-tls-init-serving-kttst\" (UID: \"93836ca5-dbd2-421f-8c04-a14b44d0da8e\") " pod="kserve/s3-tls-init-serving-kttst"
Jul 20 10:09:16.046569 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:16.046543    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2z96\" (UniqueName: \"kubernetes.io/projected/93836ca5-dbd2-421f-8c04-a14b44d0da8e-kube-api-access-w2z96\") pod \"s3-tls-init-serving-kttst\" (UID: \"93836ca5-dbd2-421f-8c04-a14b44d0da8e\") " pod="kserve/s3-tls-init-serving-kttst"
Jul 20 10:09:16.161486 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:16.161463    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-kttst"
Jul 20 10:09:16.285375 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:16.285342    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-kttst"]
Jul 20 10:09:16.287968 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:09:16.287941    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93836ca5_dbd2_421f_8c04_a14b44d0da8e.slice/crio-00fb9bc5ac54ea4a32648b76c2bc8062e26929a1d7f4b2df70d6bc9d09ff8bc3 WatchSource:0}: Error finding container 00fb9bc5ac54ea4a32648b76c2bc8062e26929a1d7f4b2df70d6bc9d09ff8bc3: Status 404 returned error can't find the container with id 00fb9bc5ac54ea4a32648b76c2bc8062e26929a1d7f4b2df70d6bc9d09ff8bc3
Jul 20 10:09:17.246533 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:17.246458    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-kttst" event={"ID":"93836ca5-dbd2-421f-8c04-a14b44d0da8e","Type":"ContainerStarted","Data":"19b62ed39c56e4be2932aac8175a6883f39092e7cb7f864c78532a6c02731437"}
Jul 20 10:09:17.246533 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:17.246513    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-kttst" event={"ID":"93836ca5-dbd2-421f-8c04-a14b44d0da8e","Type":"ContainerStarted","Data":"00fb9bc5ac54ea4a32648b76c2bc8062e26929a1d7f4b2df70d6bc9d09ff8bc3"}
Jul 20 10:09:18.251434 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:18.251400    2561 generic.go:356] "Generic (PLEG): container finished" podID="93836ca5-dbd2-421f-8c04-a14b44d0da8e" containerID="19b62ed39c56e4be2932aac8175a6883f39092e7cb7f864c78532a6c02731437" exitCode=0
Jul 20 10:09:18.251929 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:18.251475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-kttst" event={"ID":"93836ca5-dbd2-421f-8c04-a14b44d0da8e","Type":"ContainerDied","Data":"19b62ed39c56e4be2932aac8175a6883f39092e7cb7f864c78532a6c02731437"}
Jul 20 10:09:19.257350 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:19.257314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-kttst" event={"ID":"93836ca5-dbd2-421f-8c04-a14b44d0da8e","Type":"ContainerStarted","Data":"d21f00eb76323175f64ca5bf3f6e7224e28705153545e44a9280a256ae8dff93"}
Jul 20 10:09:19.277573 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:19.277534    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-kttst" podStartSLOduration=3.204309721 podStartE2EDuration="4.277518844s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.073209123s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:09:19.277203502 +0000 UTC m=+739.115302089" watchObservedRunningTime="2026-07-20 10:09:19.277518844 +0000 UTC m=+739.115617429"
Jul 20 10:09:25.283224 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:25.283188    2561 generic.go:356] "Generic (PLEG): container finished" podID="93836ca5-dbd2-421f-8c04-a14b44d0da8e" containerID="d21f00eb76323175f64ca5bf3f6e7224e28705153545e44a9280a256ae8dff93" exitCode=0
Jul 20 10:09:25.283585 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:25.283256    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-kttst" event={"ID":"93836ca5-dbd2-421f-8c04-a14b44d0da8e","Type":"ContainerDied","Data":"d21f00eb76323175f64ca5bf3f6e7224e28705153545e44a9280a256ae8dff93"}
Jul 20 10:09:26.424503 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:26.424482    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-kttst"
Jul 20 10:09:26.510861 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:26.510834    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2z96\" (UniqueName: \"kubernetes.io/projected/93836ca5-dbd2-421f-8c04-a14b44d0da8e-kube-api-access-w2z96\") pod \"93836ca5-dbd2-421f-8c04-a14b44d0da8e\" (UID: \"93836ca5-dbd2-421f-8c04-a14b44d0da8e\") "
Jul 20 10:09:26.510994 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:26.510867    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93836ca5-dbd2-421f-8c04-a14b44d0da8e-model-cache\") pod \"93836ca5-dbd2-421f-8c04-a14b44d0da8e\" (UID: \"93836ca5-dbd2-421f-8c04-a14b44d0da8e\") "
Jul 20 10:09:26.512814 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:26.512787    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93836ca5-dbd2-421f-8c04-a14b44d0da8e-kube-api-access-w2z96" (OuterVolumeSpecName: "kube-api-access-w2z96") pod "93836ca5-dbd2-421f-8c04-a14b44d0da8e" (UID: "93836ca5-dbd2-421f-8c04-a14b44d0da8e"). InnerVolumeSpecName "kube-api-access-w2z96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:09:26.535371 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:26.535343    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93836ca5-dbd2-421f-8c04-a14b44d0da8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "93836ca5-dbd2-421f-8c04-a14b44d0da8e" (UID: "93836ca5-dbd2-421f-8c04-a14b44d0da8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:09:26.611499 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:26.611479    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2z96\" (UniqueName: \"kubernetes.io/projected/93836ca5-dbd2-421f-8c04-a14b44d0da8e-kube-api-access-w2z96\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:09:26.611575 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:26.611501    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93836ca5-dbd2-421f-8c04-a14b44d0da8e-model-cache\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:09:27.291616 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:27.291587    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-kttst"
Jul 20 10:09:27.291782 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:27.291587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-kttst" event={"ID":"93836ca5-dbd2-421f-8c04-a14b44d0da8e","Type":"ContainerDied","Data":"00fb9bc5ac54ea4a32648b76c2bc8062e26929a1d7f4b2df70d6bc9d09ff8bc3"}
Jul 20 10:09:27.291782 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:27.291700    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00fb9bc5ac54ea4a32648b76c2bc8062e26929a1d7f4b2df70d6bc9d09ff8bc3"
Jul 20 10:09:51.696550 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.696513    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"]
Jul 20 10:09:51.697040 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.696761    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93836ca5-dbd2-421f-8c04-a14b44d0da8e" containerName="s3-tls-init-serving"
Jul 20 10:09:51.697040 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.696772    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="93836ca5-dbd2-421f-8c04-a14b44d0da8e" containerName="s3-tls-init-serving"
Jul 20 10:09:51.697040 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.696785    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93836ca5-dbd2-421f-8c04-a14b44d0da8e" containerName="download-hf-model"
Jul 20 10:09:51.697040 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.696790    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="93836ca5-dbd2-421f-8c04-a14b44d0da8e" containerName="download-hf-model"
Jul 20 10:09:51.697040 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.696832    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="93836ca5-dbd2-421f-8c04-a14b44d0da8e" containerName="s3-tls-init-serving"
Jul 20 10:09:51.701270 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.701250    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.711081 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.711062    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:09:51.711290 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.711273    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 20 10:09:51.711350 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.711287    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:09:51.711405 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.711360    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:09:51.711551 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.711536    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 20 10:09:51.711629 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.711611    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:09:51.715353 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.715339    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:09:51.744807 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.744781    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"]
Jul 20 10:09:51.790398 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.790374    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.790534 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.790405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70cd0f26-90ba-44cf-b4e6-aa93b423410c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.790534 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.790486    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.790635 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.790550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwg78\" (UniqueName: \"kubernetes.io/projected/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kube-api-access-jwg78\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.790635 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.790579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70cd0f26-90ba-44cf-b4e6-aa93b423410c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.891098 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.891062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwg78\" (UniqueName: \"kubernetes.io/projected/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kube-api-access-jwg78\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.891283 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.891142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70cd0f26-90ba-44cf-b4e6-aa93b423410c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.891283 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.891198    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.891283 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.891226    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70cd0f26-90ba-44cf-b4e6-aa93b423410c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.891283 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.891274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.891677 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.891647    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.891939 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.891915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70cd0f26-90ba-44cf-b4e6-aa93b423410c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.892029 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.891977    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.893816 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.893797    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70cd0f26-90ba-44cf-b4e6-aa93b423410c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:51.902018 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:51.901995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwg78\" (UniqueName: \"kubernetes.io/projected/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kube-api-access-jwg78\") pod \"isvc-sklearn-batcher-predictor-99c7db5b8-phrrr\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:52.009993 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:52.009928    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:09:52.142290 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:52.142265    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"]
Jul 20 10:09:52.144690 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:09:52.144663    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cd0f26_90ba_44cf_b4e6_aa93b423410c.slice/crio-66864ba5d43c0b515b0bf466333a07c48a55c26e9c65f20a0133925a891b1482 WatchSource:0}: Error finding container 66864ba5d43c0b515b0bf466333a07c48a55c26e9c65f20a0133925a891b1482: Status 404 returned error can't find the container with id 66864ba5d43c0b515b0bf466333a07c48a55c26e9c65f20a0133925a891b1482
Jul 20 10:09:52.380716 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:52.380657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" event={"ID":"70cd0f26-90ba-44cf-b4e6-aa93b423410c","Type":"ContainerStarted","Data":"5f373aa1b12e5434b5e003b0086c341fa00be101ab760830bf7b002a504d20c7"}
Jul 20 10:09:52.380877 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:52.380768    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" event={"ID":"70cd0f26-90ba-44cf-b4e6-aa93b423410c","Type":"ContainerStarted","Data":"66864ba5d43c0b515b0bf466333a07c48a55c26e9c65f20a0133925a891b1482"}
Jul 20 10:09:56.397814 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:56.397785    2561 generic.go:356] "Generic (PLEG): container finished" podID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerID="5f373aa1b12e5434b5e003b0086c341fa00be101ab760830bf7b002a504d20c7" exitCode=0
Jul 20 10:09:56.398213 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:09:56.397852    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" event={"ID":"70cd0f26-90ba-44cf-b4e6-aa93b423410c","Type":"ContainerDied","Data":"5f373aa1b12e5434b5e003b0086c341fa00be101ab760830bf7b002a504d20c7"}
Jul 20 10:10:09.472745 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:09.472714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" event={"ID":"70cd0f26-90ba-44cf-b4e6-aa93b423410c","Type":"ContainerStarted","Data":"f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf"}
Jul 20 10:10:12.489266 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:12.489214    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" event={"ID":"70cd0f26-90ba-44cf-b4e6-aa93b423410c","Type":"ContainerStarted","Data":"7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332"}
Jul 20 10:10:14.499618 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:14.499582    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" event={"ID":"70cd0f26-90ba-44cf-b4e6-aa93b423410c","Type":"ContainerStarted","Data":"c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de"}
Jul 20 10:10:14.500001 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:14.499772    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:10:14.500001 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:14.499932    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:10:14.500999 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:14.500973    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:10:14.526085 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:14.526040    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podStartSLOduration=2.541511953 podStartE2EDuration="23.526028674s" totalImagesPullingTime="17.284866901s" totalInitContainerRuntime="3.69964982s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:09:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:10:14.524565935 +0000 UTC m=+794.362664544" watchObservedRunningTime="2026-07-20 10:10:14.526028674 +0000 UTC m=+794.364127259"
Jul 20 10:10:15.503428 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:15.503390    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:10:15.503855 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:15.503455    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:10:15.504471 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:15.504446    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:10:16.507130 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:16.507086    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:10:16.507696 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:16.507653    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:10:16.511007 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:16.510986    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:10:17.510751 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:17.510711    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:10:17.511146 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:17.511022    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:10:27.510719 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:27.510633    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:10:27.511222 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:27.511017    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:10:37.510890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:37.510848    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:10:37.511400 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:37.511371    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:10:47.511162 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:47.511041    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:10:47.511523 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:47.511492    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:10:57.511014 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:57.510961    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:10:57.511433 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:10:57.511409    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:11:07.510664 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:07.510622    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:11:07.511096 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:07.510964    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:11:17.511077 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:17.511032    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:11:17.511542 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:17.511514    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:11:27.512093 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:27.512058    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:11:27.512778 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:27.512350    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:11:36.951303 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:36.951271    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"]
Jul 20 10:11:36.951741 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:36.951508    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" containerID="cri-o://f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf" gracePeriod=30
Jul 20 10:11:36.951741 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:36.951530    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" containerID="cri-o://c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de" gracePeriod=30
Jul 20 10:11:36.951741 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:36.951569    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kube-rbac-proxy" containerID="cri-o://7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332" gracePeriod=30
Jul 20 10:11:36.984863 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:36.984834    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"]
Jul 20 10:11:36.988410 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:36.988391    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:36.990932 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:36.990913    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 20 10:11:36.991056 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:36.990972    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 20 10:11:37.000948 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.000928    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"]
Jul 20 10:11:37.060330 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.060297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.060479 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.060343    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.060479 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.060370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.060479 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.060418    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmtng\" (UniqueName: \"kubernetes.io/projected/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kube-api-access-bmtng\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.060479 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.060467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.161620 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.161585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.161620 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.161626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmtng\" (UniqueName: \"kubernetes.io/projected/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kube-api-access-bmtng\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.161873 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.161654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.161873 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.161688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.161873 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.161730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.162187 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.162160    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.162433 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.162409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.162508 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.162458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.164242 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.164218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.170787 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.170768    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmtng\" (UniqueName: \"kubernetes.io/projected/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kube-api-access-bmtng\") pod \"isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.297684 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.297597    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:37.430833 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.430288    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"]
Jul 20 10:11:37.510768 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.510733    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:11:37.511156 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.511132    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:11:37.804653 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.804624    2561 generic.go:356] "Generic (PLEG): container finished" podID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerID="7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332" exitCode=2
Jul 20 10:11:37.804797 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.804697    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" event={"ID":"70cd0f26-90ba-44cf-b4e6-aa93b423410c","Type":"ContainerDied","Data":"7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332"}
Jul 20 10:11:37.806224 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.806198    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" event={"ID":"9b60bcbc-279f-4033-b37c-fb038c2d6bc9","Type":"ContainerStarted","Data":"a091a0e5d227a8c36efe27ef549e00e9f4f4ef96f436e5f17c4581a9e733c624"}
Jul 20 10:11:37.806327 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:37.806251    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" event={"ID":"9b60bcbc-279f-4033-b37c-fb038c2d6bc9","Type":"ContainerStarted","Data":"2d28a607f5f329c1f0b3d58c6cceeb5fa20e4117ad5eb271c8e74ccad5324620"}
Jul 20 10:11:41.508040 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:41.507969    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 20 10:11:41.823658 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:41.823583    2561 generic.go:356] "Generic (PLEG): container finished" podID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerID="f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf" exitCode=0
Jul 20 10:11:41.823810 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:41.823660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" event={"ID":"70cd0f26-90ba-44cf-b4e6-aa93b423410c","Type":"ContainerDied","Data":"f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf"}
Jul 20 10:11:41.827133 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:41.827091    2561 generic.go:356] "Generic (PLEG): container finished" podID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerID="a091a0e5d227a8c36efe27ef549e00e9f4f4ef96f436e5f17c4581a9e733c624" exitCode=0
Jul 20 10:11:41.827275 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:41.827144    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" event={"ID":"9b60bcbc-279f-4033-b37c-fb038c2d6bc9","Type":"ContainerDied","Data":"a091a0e5d227a8c36efe27ef549e00e9f4f4ef96f436e5f17c4581a9e733c624"}
Jul 20 10:11:42.833294 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:42.833260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" event={"ID":"9b60bcbc-279f-4033-b37c-fb038c2d6bc9","Type":"ContainerStarted","Data":"bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab"}
Jul 20 10:11:42.833726 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:42.833332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" event={"ID":"9b60bcbc-279f-4033-b37c-fb038c2d6bc9","Type":"ContainerStarted","Data":"37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d"}
Jul 20 10:11:42.833726 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:42.833356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" event={"ID":"9b60bcbc-279f-4033-b37c-fb038c2d6bc9","Type":"ContainerStarted","Data":"bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e"}
Jul 20 10:11:42.833726 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:42.833640    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:42.833726 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:42.833695    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:42.833726 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:42.833719    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:42.835301 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:42.835270    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 20 10:11:42.835955 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:42.835931    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:11:42.868015 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:42.867975    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podStartSLOduration=3.011315493 podStartE2EDuration="6.867962283s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.85664679s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:11:42.867373464 +0000 UTC m=+882.705472050" watchObservedRunningTime="2026-07-20 10:11:42.867962283 +0000 UTC m=+882.706060867"
Jul 20 10:11:43.837949 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:43.837905    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 20 10:11:43.838445 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:43.838423    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:11:46.507923 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:46.507881    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 20 10:11:47.511668 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:47.511566    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:11:47.512067 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:47.511904    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:11:48.842833 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:48.842806    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:11:48.843395 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:48.843357    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 20 10:11:48.843627 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:48.843604    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:11:51.507657 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:51.507620    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 20 10:11:51.508029 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:51.507732    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:11:56.507304 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:56.507220    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 20 10:11:57.511084 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:57.511037    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 20 10:11:57.511482 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:57.511226    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:11:57.511482 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:57.511308    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:11:57.511482 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:57.511380    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:11:58.844074 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:58.844032    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 20 10:11:58.844569 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:11:58.844430    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:12:00.648378 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:00.648349    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:12:00.653629 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:00.653608    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:12:01.508131 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:01.508083    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 20 10:12:06.507503 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:06.507461    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 20 10:12:07.088310 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.088286    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:12:07.189098 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.189066    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwg78\" (UniqueName: \"kubernetes.io/projected/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kube-api-access-jwg78\") pod \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") "
Jul 20 10:12:07.189266 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.189106    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kserve-provision-location\") pod \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") "
Jul 20 10:12:07.189266 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.189241    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70cd0f26-90ba-44cf-b4e6-aa93b423410c-proxy-tls\") pod \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") "
Jul 20 10:12:07.189355 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.189272    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kube-rbac-proxy-sar-config\") pod \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") "
Jul 20 10:12:07.189355 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.189314    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70cd0f26-90ba-44cf-b4e6-aa93b423410c-cabundle-cert\") pod \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\" (UID: \"70cd0f26-90ba-44cf-b4e6-aa93b423410c\") "
Jul 20 10:12:07.189456 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.189430    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70cd0f26-90ba-44cf-b4e6-aa93b423410c" (UID: "70cd0f26-90ba-44cf-b4e6-aa93b423410c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:12:07.189611 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.189591    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:12:07.189678 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.189641    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "70cd0f26-90ba-44cf-b4e6-aa93b423410c" (UID: "70cd0f26-90ba-44cf-b4e6-aa93b423410c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:12:07.189760 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.189741    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70cd0f26-90ba-44cf-b4e6-aa93b423410c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70cd0f26-90ba-44cf-b4e6-aa93b423410c" (UID: "70cd0f26-90ba-44cf-b4e6-aa93b423410c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:12:07.191290 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.191270    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kube-api-access-jwg78" (OuterVolumeSpecName: "kube-api-access-jwg78") pod "70cd0f26-90ba-44cf-b4e6-aa93b423410c" (UID: "70cd0f26-90ba-44cf-b4e6-aa93b423410c"). InnerVolumeSpecName "kube-api-access-jwg78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:12:07.191410 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.191395    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70cd0f26-90ba-44cf-b4e6-aa93b423410c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70cd0f26-90ba-44cf-b4e6-aa93b423410c" (UID: "70cd0f26-90ba-44cf-b4e6-aa93b423410c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:12:07.290428 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.290354    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwg78\" (UniqueName: \"kubernetes.io/projected/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kube-api-access-jwg78\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:12:07.290428 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.290380    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70cd0f26-90ba-44cf-b4e6-aa93b423410c-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:12:07.290428 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.290391    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70cd0f26-90ba-44cf-b4e6-aa93b423410c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:12:07.290428 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.290400    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70cd0f26-90ba-44cf-b4e6-aa93b423410c-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:12:07.931079 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.931045    2561 generic.go:356] "Generic (PLEG): container finished" podID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerID="c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de" exitCode=0
Jul 20 10:12:07.931457 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.931103    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"
Jul 20 10:12:07.931457 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.931135    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" event={"ID":"70cd0f26-90ba-44cf-b4e6-aa93b423410c","Type":"ContainerDied","Data":"c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de"}
Jul 20 10:12:07.931457 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.931201    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr" event={"ID":"70cd0f26-90ba-44cf-b4e6-aa93b423410c","Type":"ContainerDied","Data":"66864ba5d43c0b515b0bf466333a07c48a55c26e9c65f20a0133925a891b1482"}
Jul 20 10:12:07.931457 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.931231    2561 scope.go:117] "RemoveContainer" containerID="c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de"
Jul 20 10:12:07.939432 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.939419    2561 scope.go:117] "RemoveContainer" containerID="7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332"
Jul 20 10:12:07.946220 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.946205    2561 scope.go:117] "RemoveContainer" containerID="f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf"
Jul 20 10:12:07.953155 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.953136    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"]
Jul 20 10:12:07.953378 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.953362    2561 scope.go:117] "RemoveContainer" containerID="5f373aa1b12e5434b5e003b0086c341fa00be101ab760830bf7b002a504d20c7"
Jul 20 10:12:07.957013 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.956991    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-99c7db5b8-phrrr"]
Jul 20 10:12:07.960851 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.960699    2561 scope.go:117] "RemoveContainer" containerID="c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de"
Jul 20 10:12:07.960988 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:12:07.960968    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de\": container with ID starting with c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de not found: ID does not exist" containerID="c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de"
Jul 20 10:12:07.961031 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.960997    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de"} err="failed to get container status \"c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de\": rpc error: code = NotFound desc = could not find container \"c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de\": container with ID starting with c24450d2cc7d90cfb5c1799e5fe7beeb12bcec92cfdab97a2e06be1e48e615de not found: ID does not exist"
Jul 20 10:12:07.961031 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.961013    2561 scope.go:117] "RemoveContainer" containerID="7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332"
Jul 20 10:12:07.961284 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:12:07.961266    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332\": container with ID starting with 7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332 not found: ID does not exist" containerID="7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332"
Jul 20 10:12:07.961342 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.961290    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332"} err="failed to get container status \"7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332\": rpc error: code = NotFound desc = could not find container \"7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332\": container with ID starting with 7fe78a50606cbe548770d416e1492338e4d3eec6b83f2bac1981508384530332 not found: ID does not exist"
Jul 20 10:12:07.961342 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.961306    2561 scope.go:117] "RemoveContainer" containerID="f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf"
Jul 20 10:12:07.961566 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:12:07.961549    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf\": container with ID starting with f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf not found: ID does not exist" containerID="f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf"
Jul 20 10:12:07.961613 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.961569    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf"} err="failed to get container status \"f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf\": rpc error: code = NotFound desc = could not find container \"f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf\": container with ID starting with f7263ac03f312982bf3c554596ffcc7415fd0830fc07f42b2e697ca46d72f2cf not found: ID does not exist"
Jul 20 10:12:07.961613 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.961584    2561 scope.go:117] "RemoveContainer" containerID="5f373aa1b12e5434b5e003b0086c341fa00be101ab760830bf7b002a504d20c7"
Jul 20 10:12:07.961856 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:12:07.961841    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f373aa1b12e5434b5e003b0086c341fa00be101ab760830bf7b002a504d20c7\": container with ID starting with 5f373aa1b12e5434b5e003b0086c341fa00be101ab760830bf7b002a504d20c7 not found: ID does not exist" containerID="5f373aa1b12e5434b5e003b0086c341fa00be101ab760830bf7b002a504d20c7"
Jul 20 10:12:07.961896 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:07.961858    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f373aa1b12e5434b5e003b0086c341fa00be101ab760830bf7b002a504d20c7"} err="failed to get container status \"5f373aa1b12e5434b5e003b0086c341fa00be101ab760830bf7b002a504d20c7\": rpc error: code = NotFound desc = could not find container \"5f373aa1b12e5434b5e003b0086c341fa00be101ab760830bf7b002a504d20c7\": container with ID starting with 5f373aa1b12e5434b5e003b0086c341fa00be101ab760830bf7b002a504d20c7 not found: ID does not exist"
Jul 20 10:12:08.765682 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:08.765654    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" path="/var/lib/kubelet/pods/70cd0f26-90ba-44cf-b4e6-aa93b423410c/volumes"
Jul 20 10:12:08.843758 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:08.843722    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 20 10:12:08.844173 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:08.844152    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:12:18.843944 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:18.843904    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 20 10:12:18.844459 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:18.844438    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:12:28.843318 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:28.843279    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 20 10:12:28.843690 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:28.843658    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:12:38.844034 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:38.843995    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 20 10:12:38.844480 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:38.844457    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:12:48.844038 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:48.843994    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 20 10:12:48.844474 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:48.844423    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:12:58.843773 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:58.843740    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:12:58.844275 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:12:58.843956    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:13:12.078929 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.078899    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"]
Jul 20 10:13:12.079313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.079197    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="storage-initializer"
Jul 20 10:13:12.079313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.079210    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="storage-initializer"
Jul 20 10:13:12.079313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.079221    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container"
Jul 20 10:13:12.079313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.079228    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container"
Jul 20 10:13:12.079313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.079236    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kube-rbac-proxy"
Jul 20 10:13:12.079313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.079242    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kube-rbac-proxy"
Jul 20 10:13:12.079313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.079249    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent"
Jul 20 10:13:12.079313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.079255    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent"
Jul 20 10:13:12.079313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.079294    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="agent"
Jul 20 10:13:12.079313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.079301    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kserve-container"
Jul 20 10:13:12.079313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.079309    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="70cd0f26-90ba-44cf-b4e6-aa93b423410c" containerName="kube-rbac-proxy"
Jul 20 10:13:12.083285 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.083262    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:12.085947 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.085924    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 20 10:13:12.086069 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.085924    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 20 10:13:12.097424 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.097402    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"]
Jul 20 10:13:12.100989 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.100967    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"]
Jul 20 10:13:12.101271 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.101228    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" containerID="cri-o://bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e" gracePeriod=30
Jul 20 10:13:12.101320 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.101278    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" containerID="cri-o://bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab" gracePeriod=30
Jul 20 10:13:12.101355 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.101320    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kube-rbac-proxy" containerID="cri-o://37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d" gracePeriod=30
Jul 20 10:13:12.139698 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.139674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feb64b36-d3b0-4d82-859a-a0aca37a4690-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2cjh5\" (UID: \"feb64b36-d3b0-4d82-859a-a0aca37a4690\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:12.139825 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.139717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgb2\" (UniqueName: \"kubernetes.io/projected/feb64b36-d3b0-4d82-859a-a0aca37a4690-kube-api-access-ttgb2\") pod \"message-dumper-predictor-6575bcfdfd-2cjh5\" (UID: \"feb64b36-d3b0-4d82-859a-a0aca37a4690\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:12.139825 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.139808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feb64b36-d3b0-4d82-859a-a0aca37a4690-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-2cjh5\" (UID: \"feb64b36-d3b0-4d82-859a-a0aca37a4690\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:12.240796 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.240767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feb64b36-d3b0-4d82-859a-a0aca37a4690-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2cjh5\" (UID: \"feb64b36-d3b0-4d82-859a-a0aca37a4690\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:12.240940 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.240805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgb2\" (UniqueName: \"kubernetes.io/projected/feb64b36-d3b0-4d82-859a-a0aca37a4690-kube-api-access-ttgb2\") pod \"message-dumper-predictor-6575bcfdfd-2cjh5\" (UID: \"feb64b36-d3b0-4d82-859a-a0aca37a4690\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:12.240940 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.240841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feb64b36-d3b0-4d82-859a-a0aca37a4690-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-2cjh5\" (UID: \"feb64b36-d3b0-4d82-859a-a0aca37a4690\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:12.241422 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.241398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feb64b36-d3b0-4d82-859a-a0aca37a4690-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-2cjh5\" (UID: \"feb64b36-d3b0-4d82-859a-a0aca37a4690\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:12.243226 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.243201    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feb64b36-d3b0-4d82-859a-a0aca37a4690-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2cjh5\" (UID: \"feb64b36-d3b0-4d82-859a-a0aca37a4690\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:12.250619 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.250597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgb2\" (UniqueName: \"kubernetes.io/projected/feb64b36-d3b0-4d82-859a-a0aca37a4690-kube-api-access-ttgb2\") pod \"message-dumper-predictor-6575bcfdfd-2cjh5\" (UID: \"feb64b36-d3b0-4d82-859a-a0aca37a4690\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:12.392652 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.392620    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:12.538009 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.537974    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"]
Jul 20 10:13:12.540904 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:13:12.540878    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb64b36_d3b0_4d82_859a_a0aca37a4690.slice/crio-7feb65edd4b3ff09f950f264d7d25ed826bca8c63a68555d73ea040558e07541 WatchSource:0}: Error finding container 7feb65edd4b3ff09f950f264d7d25ed826bca8c63a68555d73ea040558e07541: Status 404 returned error can't find the container with id 7feb65edd4b3ff09f950f264d7d25ed826bca8c63a68555d73ea040558e07541
Jul 20 10:13:12.542724 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:12.542708    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:13:13.178102 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:13.178065    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5" event={"ID":"feb64b36-d3b0-4d82-859a-a0aca37a4690","Type":"ContainerStarted","Data":"7feb65edd4b3ff09f950f264d7d25ed826bca8c63a68555d73ea040558e07541"}
Jul 20 10:13:13.180424 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:13.180402    2561 generic.go:356] "Generic (PLEG): container finished" podID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerID="37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d" exitCode=2
Jul 20 10:13:13.180538 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:13.180431    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" event={"ID":"9b60bcbc-279f-4033-b37c-fb038c2d6bc9","Type":"ContainerDied","Data":"37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d"}
Jul 20 10:13:13.838361 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:13.838332    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 20 10:13:14.185873 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:14.185836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5" event={"ID":"feb64b36-d3b0-4d82-859a-a0aca37a4690","Type":"ContainerStarted","Data":"82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e"}
Jul 20 10:13:14.186296 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:14.185930    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:14.186296 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:14.185962    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5" event={"ID":"feb64b36-d3b0-4d82-859a-a0aca37a4690","Type":"ContainerStarted","Data":"da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a"}
Jul 20 10:13:14.186296 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:14.186136    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:14.187682 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:14.187662    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:14.253021 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:14.252974    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5" podStartSLOduration=1.044760565 podStartE2EDuration="2.252963184s" totalImagesPullingTime="1.208202619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:13:14.225521984 +0000 UTC m=+974.063620569" watchObservedRunningTime="2026-07-20 10:13:14.252963184 +0000 UTC m=+974.091061801"
Jul 20 10:13:17.199552 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:17.199516    2561 generic.go:356] "Generic (PLEG): container finished" podID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerID="bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e" exitCode=0
Jul 20 10:13:17.199907 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:17.199584    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" event={"ID":"9b60bcbc-279f-4033-b37c-fb038c2d6bc9","Type":"ContainerDied","Data":"bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e"}
Jul 20 10:13:18.838983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:18.838939    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 20 10:13:18.843315 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:18.843288    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 20 10:13:18.843616 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:18.843592    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:13:21.198877 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:21.198843    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:13:22.141702 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.141663    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"]
Jul 20 10:13:22.145152 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.145109    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.149929 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.149909    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 20 10:13:22.150289 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.150274    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 20 10:13:22.168925 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.168901    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"]
Jul 20 10:13:22.202966 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.202939    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7b3b6fb-7783-4df3-942b-e80dbb42445c-cabundle-cert\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.203287 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.202987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kserve-provision-location\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.203287 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.203020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzgp9\" (UniqueName: \"kubernetes.io/projected/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kube-api-access-gzgp9\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.203287 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.203070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7b3b6fb-7783-4df3-942b-e80dbb42445c-proxy-tls\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.203287 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.203136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.303992 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.303962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kserve-provision-location\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.303992 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.303996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgp9\" (UniqueName: \"kubernetes.io/projected/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kube-api-access-gzgp9\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.304217 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.304144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7b3b6fb-7783-4df3-942b-e80dbb42445c-proxy-tls\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.304271 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.304215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.304326 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.304266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7b3b6fb-7783-4df3-942b-e80dbb42445c-cabundle-cert\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.304425 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.304406    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kserve-provision-location\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.304905 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.304883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.304991 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.304888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7b3b6fb-7783-4df3-942b-e80dbb42445c-cabundle-cert\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.306659 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.306637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7b3b6fb-7783-4df3-942b-e80dbb42445c-proxy-tls\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.314404 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.314381    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgp9\" (UniqueName: \"kubernetes.io/projected/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kube-api-access-gzgp9\") pod \"isvc-logger-predictor-5496cb9695-v2lg2\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.455733 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.455669    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:22.589272 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:22.589241    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"]
Jul 20 10:13:22.592240 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:13:22.592207    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b3b6fb_7783_4df3_942b_e80dbb42445c.slice/crio-0603e2f0691b25348f5f1b63bc75a00766a3eacb2e3533db17a5b0153ff36d3e WatchSource:0}: Error finding container 0603e2f0691b25348f5f1b63bc75a00766a3eacb2e3533db17a5b0153ff36d3e: Status 404 returned error can't find the container with id 0603e2f0691b25348f5f1b63bc75a00766a3eacb2e3533db17a5b0153ff36d3e
Jul 20 10:13:23.221611 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:23.221578    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" event={"ID":"a7b3b6fb-7783-4df3-942b-e80dbb42445c","Type":"ContainerStarted","Data":"6c97cc83074f743feceeb2d8e49c5f59077d9aa35fb3fd8cee28a1e7b7dd62d7"}
Jul 20 10:13:23.221998 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:23.221644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" event={"ID":"a7b3b6fb-7783-4df3-942b-e80dbb42445c","Type":"ContainerStarted","Data":"0603e2f0691b25348f5f1b63bc75a00766a3eacb2e3533db17a5b0153ff36d3e"}
Jul 20 10:13:23.838871 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:23.838833    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 20 10:13:23.839027 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:23.838958    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:13:26.234495 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:26.234467    2561 generic.go:356] "Generic (PLEG): container finished" podID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerID="6c97cc83074f743feceeb2d8e49c5f59077d9aa35fb3fd8cee28a1e7b7dd62d7" exitCode=0
Jul 20 10:13:26.234840 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:26.234537    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" event={"ID":"a7b3b6fb-7783-4df3-942b-e80dbb42445c","Type":"ContainerDied","Data":"6c97cc83074f743feceeb2d8e49c5f59077d9aa35fb3fd8cee28a1e7b7dd62d7"}
Jul 20 10:13:27.241918 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:27.241882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" event={"ID":"a7b3b6fb-7783-4df3-942b-e80dbb42445c","Type":"ContainerStarted","Data":"d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271"}
Jul 20 10:13:27.242319 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:27.241955    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" event={"ID":"a7b3b6fb-7783-4df3-942b-e80dbb42445c","Type":"ContainerStarted","Data":"3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd"}
Jul 20 10:13:27.242319 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:27.241982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" event={"ID":"a7b3b6fb-7783-4df3-942b-e80dbb42445c","Type":"ContainerStarted","Data":"8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671"}
Jul 20 10:13:27.242319 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:27.242223    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:27.242319 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:27.242282    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:27.243585 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:27.243557    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:13:27.271206 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:27.271163    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podStartSLOduration=1.715696176 podStartE2EDuration="5.27114551s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.555449334s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:13:27.268899388 +0000 UTC m=+987.106997996" watchObservedRunningTime="2026-07-20 10:13:27.27114551 +0000 UTC m=+987.109244100"
Jul 20 10:13:28.245655 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:28.245621    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:28.246137 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:28.245766    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:13:28.246898 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:28.246876    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:13:28.838490 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:28.838454    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 20 10:13:28.844252 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:28.844229    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 20 10:13:28.844627 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:28.844605    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:13:29.249336 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:29.249301    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:13:29.249803 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:29.249776    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:13:33.838389 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:33.838341    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 20 10:13:34.255132 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:34.255096    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:13:34.255675 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:34.255648    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:13:34.255983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:34.255956    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:13:38.838278 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:38.838234    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 20 10:13:38.843572 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:38.843551    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 20 10:13:38.843674 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:38.843663    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:13:38.843813 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:38.843787    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:13:38.843922 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:38.843910    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:13:42.249524 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.249501    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:13:42.304722 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.304693    2561 generic.go:356] "Generic (PLEG): container finished" podID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerID="bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab" exitCode=0
Jul 20 10:13:42.304858 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.304749    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" event={"ID":"9b60bcbc-279f-4033-b37c-fb038c2d6bc9","Type":"ContainerDied","Data":"bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab"}
Jul 20 10:13:42.304858 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.304801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2" event={"ID":"9b60bcbc-279f-4033-b37c-fb038c2d6bc9","Type":"ContainerDied","Data":"2d28a607f5f329c1f0b3d58c6cceeb5fa20e4117ad5eb271c8e74ccad5324620"}
Jul 20 10:13:42.304858 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.304752    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"
Jul 20 10:13:42.304858 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.304832    2561 scope.go:117] "RemoveContainer" containerID="bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab"
Jul 20 10:13:42.312459 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.312443    2561 scope.go:117] "RemoveContainer" containerID="37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d"
Jul 20 10:13:42.319611 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.319593    2561 scope.go:117] "RemoveContainer" containerID="bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e"
Jul 20 10:13:42.326428 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.326412    2561 scope.go:117] "RemoveContainer" containerID="a091a0e5d227a8c36efe27ef549e00e9f4f4ef96f436e5f17c4581a9e733c624"
Jul 20 10:13:42.333123 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.333094    2561 scope.go:117] "RemoveContainer" containerID="bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab"
Jul 20 10:13:42.333405 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:13:42.333387    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab\": container with ID starting with bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab not found: ID does not exist" containerID="bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab"
Jul 20 10:13:42.333463 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.333412    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab"} err="failed to get container status \"bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab\": rpc error: code = NotFound desc = could not find container \"bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab\": container with ID starting with bde632eae2c1fdb3e9e0368018f7a6f2592c030acb93fa81a4a8b83a9f5534ab not found: ID does not exist"
Jul 20 10:13:42.333463 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.333429    2561 scope.go:117] "RemoveContainer" containerID="37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d"
Jul 20 10:13:42.333668 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:13:42.333653    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d\": container with ID starting with 37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d not found: ID does not exist" containerID="37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d"
Jul 20 10:13:42.333723 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.333672    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d"} err="failed to get container status \"37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d\": rpc error: code = NotFound desc = could not find container \"37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d\": container with ID starting with 37520a2c11589577fcb399c15665e6b6e35706a4a326910fe9ef1eebef83e86d not found: ID does not exist"
Jul 20 10:13:42.333723 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.333689    2561 scope.go:117] "RemoveContainer" containerID="bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e"
Jul 20 10:13:42.333919 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:13:42.333901    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e\": container with ID starting with bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e not found: ID does not exist" containerID="bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e"
Jul 20 10:13:42.333969 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.333926    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e"} err="failed to get container status \"bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e\": rpc error: code = NotFound desc = could not find container \"bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e\": container with ID starting with bc9a4bea86da65f29931a1a2d474de61c1bad58be473be9e86feedac84e8357e not found: ID does not exist"
Jul 20 10:13:42.333969 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.333944    2561 scope.go:117] "RemoveContainer" containerID="a091a0e5d227a8c36efe27ef549e00e9f4f4ef96f436e5f17c4581a9e733c624"
Jul 20 10:13:42.334220 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:13:42.334203    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a091a0e5d227a8c36efe27ef549e00e9f4f4ef96f436e5f17c4581a9e733c624\": container with ID starting with a091a0e5d227a8c36efe27ef549e00e9f4f4ef96f436e5f17c4581a9e733c624 not found: ID does not exist" containerID="a091a0e5d227a8c36efe27ef549e00e9f4f4ef96f436e5f17c4581a9e733c624"
Jul 20 10:13:42.334299 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.334225    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a091a0e5d227a8c36efe27ef549e00e9f4f4ef96f436e5f17c4581a9e733c624"} err="failed to get container status \"a091a0e5d227a8c36efe27ef549e00e9f4f4ef96f436e5f17c4581a9e733c624\": rpc error: code = NotFound desc = could not find container \"a091a0e5d227a8c36efe27ef549e00e9f4f4ef96f436e5f17c4581a9e733c624\": container with ID starting with a091a0e5d227a8c36efe27ef549e00e9f4f4ef96f436e5f17c4581a9e733c624 not found: ID does not exist"
Jul 20 10:13:42.358638 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.358591    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kserve-provision-location\") pod \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") "
Jul 20 10:13:42.358748 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.358650    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-cabundle-cert\") pod \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") "
Jul 20 10:13:42.358748 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.358686    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kube-rbac-proxy-sar-config\") pod \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") "
Jul 20 10:13:42.358748 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.358739    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmtng\" (UniqueName: \"kubernetes.io/projected/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kube-api-access-bmtng\") pod \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") "
Jul 20 10:13:42.358856 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.358780    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-proxy-tls\") pod \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\" (UID: \"9b60bcbc-279f-4033-b37c-fb038c2d6bc9\") "
Jul 20 10:13:42.359000 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.358909    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b60bcbc-279f-4033-b37c-fb038c2d6bc9" (UID: "9b60bcbc-279f-4033-b37c-fb038c2d6bc9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:42.359072 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.359006    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:13:42.359072 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.359004    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b60bcbc-279f-4033-b37c-fb038c2d6bc9" (UID: "9b60bcbc-279f-4033-b37c-fb038c2d6bc9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:13:42.359195 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.359084    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b60bcbc-279f-4033-b37c-fb038c2d6bc9" (UID: "9b60bcbc-279f-4033-b37c-fb038c2d6bc9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:13:42.360752 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.360728    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b60bcbc-279f-4033-b37c-fb038c2d6bc9" (UID: "9b60bcbc-279f-4033-b37c-fb038c2d6bc9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:13:42.360829 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.360780    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kube-api-access-bmtng" (OuterVolumeSpecName: "kube-api-access-bmtng") pod "9b60bcbc-279f-4033-b37c-fb038c2d6bc9" (UID: "9b60bcbc-279f-4033-b37c-fb038c2d6bc9"). InnerVolumeSpecName "kube-api-access-bmtng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:13:42.459564 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.459531    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmtng\" (UniqueName: \"kubernetes.io/projected/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kube-api-access-bmtng\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:13:42.459564 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.459557    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:13:42.459564 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.459568    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:13:42.459769 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.459577    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b60bcbc-279f-4033-b37c-fb038c2d6bc9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:13:42.637545 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.637498    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"]
Jul 20 10:13:42.639848 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.639824    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7d9cf8c775-4t5p2"]
Jul 20 10:13:42.765833 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:42.765800    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" path="/var/lib/kubelet/pods/9b60bcbc-279f-4033-b37c-fb038c2d6bc9/volumes"
Jul 20 10:13:44.255947 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:44.255899    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:13:44.256452 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:44.256387    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:13:54.255650 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:54.255601    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:13:54.256174 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:13:54.256034    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:14:04.256372 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:04.256310    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:14:04.256853 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:04.256695    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:14:14.255652 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:14.255603    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:14:14.256203 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:14.256048    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:14:24.256347 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:24.256299    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:14:24.256830 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:24.256802    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:14:34.256527 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:34.256482    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:14:34.256936 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:34.256818    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:14:44.256681 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:44.256646    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:14:44.257086 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:44.256905    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:14:57.114160 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.114129    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-2cjh5_feb64b36-d3b0-4d82-859a-a0aca37a4690/kserve-container/0.log"
Jul 20 10:14:57.511867 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.511799    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"]
Jul 20 10:14:57.512213 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.512194    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kube-rbac-proxy"
Jul 20 10:14:57.512284 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.512215    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kube-rbac-proxy"
Jul 20 10:14:57.512284 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.512229    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container"
Jul 20 10:14:57.512284 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.512236    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container"
Jul 20 10:14:57.512284 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.512249    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="storage-initializer"
Jul 20 10:14:57.512284 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.512255    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="storage-initializer"
Jul 20 10:14:57.512284 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.512260    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent"
Jul 20 10:14:57.512284 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.512265    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent"
Jul 20 10:14:57.512525 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.512319    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kube-rbac-proxy"
Jul 20 10:14:57.512525 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.512326    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="agent"
Jul 20 10:14:57.512525 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.512331    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b60bcbc-279f-4033-b37c-fb038c2d6bc9" containerName="kserve-container"
Jul 20 10:14:57.515536 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.515517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.518092 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.518069    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 20 10:14:57.518300 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.518286    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 20 10:14:57.522279 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.522255    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"]
Jul 20 10:14:57.522519 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.522484    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5" podUID="feb64b36-d3b0-4d82-859a-a0aca37a4690" containerName="kserve-container" containerID="cri-o://da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a" gracePeriod=30
Jul 20 10:14:57.522622 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.522548    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5" podUID="feb64b36-d3b0-4d82-859a-a0aca37a4690" containerName="kube-rbac-proxy" containerID="cri-o://82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e" gracePeriod=30
Jul 20 10:14:57.531443 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.531420    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"]
Jul 20 10:14:57.540055 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.540034    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"]
Jul 20 10:14:57.540418 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.540392    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" containerID="cri-o://8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671" gracePeriod=30
Jul 20 10:14:57.540494 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.540450    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" containerID="cri-o://d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271" gracePeriod=30
Jul 20 10:14:57.540494 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.540465    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kube-rbac-proxy" containerID="cri-o://3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd" gracePeriod=30
Jul 20 10:14:57.596307 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.596282    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d53fb653-f539-4be4-9dd6-7934a76cc3fe-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.596400 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.596315    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.596400 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.596331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmsmr\" (UniqueName: \"kubernetes.io/projected/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kube-api-access-gmsmr\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.596477 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.596402    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d53fb653-f539-4be4-9dd6-7934a76cc3fe-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.596477 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.596449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.697072 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.697039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d53fb653-f539-4be4-9dd6-7934a76cc3fe-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.697320 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.697083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.697320 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.697107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmsmr\" (UniqueName: \"kubernetes.io/projected/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kube-api-access-gmsmr\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.697320 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.697178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d53fb653-f539-4be4-9dd6-7934a76cc3fe-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.697320 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.697217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.697867 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.697838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d53fb653-f539-4be4-9dd6-7934a76cc3fe-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.697986 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.697572    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.698457 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.698432    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.700143 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.700093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d53fb653-f539-4be4-9dd6-7934a76cc3fe-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.707377 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.707354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmsmr\" (UniqueName: \"kubernetes.io/projected/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kube-api-access-gmsmr\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bp2w5\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.765617 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.765567    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:14:57.798269 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.798245    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feb64b36-d3b0-4d82-859a-a0aca37a4690-proxy-tls\") pod \"feb64b36-d3b0-4d82-859a-a0aca37a4690\" (UID: \"feb64b36-d3b0-4d82-859a-a0aca37a4690\") "
Jul 20 10:14:57.798383 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.798290    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttgb2\" (UniqueName: \"kubernetes.io/projected/feb64b36-d3b0-4d82-859a-a0aca37a4690-kube-api-access-ttgb2\") pod \"feb64b36-d3b0-4d82-859a-a0aca37a4690\" (UID: \"feb64b36-d3b0-4d82-859a-a0aca37a4690\") "
Jul 20 10:14:57.798383 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.798375    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feb64b36-d3b0-4d82-859a-a0aca37a4690-kube-rbac-proxy-sar-config\") pod \"feb64b36-d3b0-4d82-859a-a0aca37a4690\" (UID: \"feb64b36-d3b0-4d82-859a-a0aca37a4690\") "
Jul 20 10:14:57.798725 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.798701    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb64b36-d3b0-4d82-859a-a0aca37a4690-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "feb64b36-d3b0-4d82-859a-a0aca37a4690" (UID: "feb64b36-d3b0-4d82-859a-a0aca37a4690"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:14:57.800093 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.800074    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb64b36-d3b0-4d82-859a-a0aca37a4690-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "feb64b36-d3b0-4d82-859a-a0aca37a4690" (UID: "feb64b36-d3b0-4d82-859a-a0aca37a4690"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:14:57.800210 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.800192    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb64b36-d3b0-4d82-859a-a0aca37a4690-kube-api-access-ttgb2" (OuterVolumeSpecName: "kube-api-access-ttgb2") pod "feb64b36-d3b0-4d82-859a-a0aca37a4690" (UID: "feb64b36-d3b0-4d82-859a-a0aca37a4690"). InnerVolumeSpecName "kube-api-access-ttgb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:14:57.828385 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.828352    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:14:57.898944 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.898904    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feb64b36-d3b0-4d82-859a-a0aca37a4690-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:14:57.898944 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.898931    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feb64b36-d3b0-4d82-859a-a0aca37a4690-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:14:57.898944 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.898947    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgb2\" (UniqueName: \"kubernetes.io/projected/feb64b36-d3b0-4d82-859a-a0aca37a4690-kube-api-access-ttgb2\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:14:57.965253 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:57.965221    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"]
Jul 20 10:14:57.968287 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:14:57.968260    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53fb653_f539_4be4_9dd6_7934a76cc3fe.slice/crio-5f160af285b0310f43200db25053350b26596c025292d7a2cf4181251aceb7e5 WatchSource:0}: Error finding container 5f160af285b0310f43200db25053350b26596c025292d7a2cf4181251aceb7e5: Status 404 returned error can't find the container with id 5f160af285b0310f43200db25053350b26596c025292d7a2cf4181251aceb7e5
Jul 20 10:14:58.588267 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.588236    2561 generic.go:356] "Generic (PLEG): container finished" podID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerID="3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd" exitCode=2
Jul 20 10:14:58.588670 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.588314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" event={"ID":"a7b3b6fb-7783-4df3-942b-e80dbb42445c","Type":"ContainerDied","Data":"3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd"}
Jul 20 10:14:58.589643 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.589623    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" event={"ID":"d53fb653-f539-4be4-9dd6-7934a76cc3fe","Type":"ContainerStarted","Data":"2bdac40b66e47fa0579e92a9a44df2d9458ccd0c5c36f72b45bfc52d55094a56"}
Jul 20 10:14:58.589742 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.589679    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" event={"ID":"d53fb653-f539-4be4-9dd6-7934a76cc3fe","Type":"ContainerStarted","Data":"5f160af285b0310f43200db25053350b26596c025292d7a2cf4181251aceb7e5"}
Jul 20 10:14:58.591043 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.591022    2561 generic.go:356] "Generic (PLEG): container finished" podID="feb64b36-d3b0-4d82-859a-a0aca37a4690" containerID="82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e" exitCode=2
Jul 20 10:14:58.591043 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.591039    2561 generic.go:356] "Generic (PLEG): container finished" podID="feb64b36-d3b0-4d82-859a-a0aca37a4690" containerID="da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a" exitCode=2
Jul 20 10:14:58.591181 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.591067    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"
Jul 20 10:14:58.591181 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.591076    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5" event={"ID":"feb64b36-d3b0-4d82-859a-a0aca37a4690","Type":"ContainerDied","Data":"82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e"}
Jul 20 10:14:58.591181 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.591130    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5" event={"ID":"feb64b36-d3b0-4d82-859a-a0aca37a4690","Type":"ContainerDied","Data":"da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a"}
Jul 20 10:14:58.591181 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.591150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5" event={"ID":"feb64b36-d3b0-4d82-859a-a0aca37a4690","Type":"ContainerDied","Data":"7feb65edd4b3ff09f950f264d7d25ed826bca8c63a68555d73ea040558e07541"}
Jul 20 10:14:58.591181 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.591174    2561 scope.go:117] "RemoveContainer" containerID="82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e"
Jul 20 10:14:58.599432 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.599414    2561 scope.go:117] "RemoveContainer" containerID="da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a"
Jul 20 10:14:58.606816 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.606802    2561 scope.go:117] "RemoveContainer" containerID="82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e"
Jul 20 10:14:58.607046 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:14:58.607030    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e\": container with ID starting with 82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e not found: ID does not exist" containerID="82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e"
Jul 20 10:14:58.607086 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.607055    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e"} err="failed to get container status \"82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e\": rpc error: code = NotFound desc = could not find container \"82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e\": container with ID starting with 82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e not found: ID does not exist"
Jul 20 10:14:58.607086 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.607071    2561 scope.go:117] "RemoveContainer" containerID="da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a"
Jul 20 10:14:58.607350 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:14:58.607332    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a\": container with ID starting with da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a not found: ID does not exist" containerID="da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a"
Jul 20 10:14:58.607398 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.607356    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a"} err="failed to get container status \"da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a\": rpc error: code = NotFound desc = could not find container \"da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a\": container with ID starting with da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a not found: ID does not exist"
Jul 20 10:14:58.607398 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.607370    2561 scope.go:117] "RemoveContainer" containerID="82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e"
Jul 20 10:14:58.607601 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.607584    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e"} err="failed to get container status \"82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e\": rpc error: code = NotFound desc = could not find container \"82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e\": container with ID starting with 82622a5ac6da8ba382586d9869f5f433c12d95f4388c6e4060edb45a67f10b9e not found: ID does not exist"
Jul 20 10:14:58.607681 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.607603    2561 scope.go:117] "RemoveContainer" containerID="da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a"
Jul 20 10:14:58.607819 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.607804    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a"} err="failed to get container status \"da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a\": rpc error: code = NotFound desc = could not find container \"da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a\": container with ID starting with da31cdf6ab33ad491fe78ce8b5d17c2b48aedae1f0567547ebc08e896b27af7a not found: ID does not exist"
Jul 20 10:14:58.641329 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.641306    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"]
Jul 20 10:14:58.644730 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.644710    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2cjh5"]
Jul 20 10:14:58.765743 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:58.765717    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feb64b36-d3b0-4d82-859a-a0aca37a4690" path="/var/lib/kubelet/pods/feb64b36-d3b0-4d82-859a-a0aca37a4690/volumes"
Jul 20 10:14:59.250129 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:14:59.250077    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 20 10:15:02.612129 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:02.612073    2561 generic.go:356] "Generic (PLEG): container finished" podID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerID="8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671" exitCode=0
Jul 20 10:15:02.612487 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:02.612146    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" event={"ID":"a7b3b6fb-7783-4df3-942b-e80dbb42445c","Type":"ContainerDied","Data":"8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671"}
Jul 20 10:15:04.249759 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:04.249719    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 20 10:15:04.255950 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:04.255920    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:15:04.256252 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:04.256229    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:15:09.249691 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:09.249647    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 20 10:15:09.250056 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:09.249777    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:15:14.249655 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:14.249611    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 20 10:15:14.256028 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:14.255995    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:15:14.256384 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:14.256360    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:15:18.673657 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:18.673580    2561 generic.go:356] "Generic (PLEG): container finished" podID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerID="2bdac40b66e47fa0579e92a9a44df2d9458ccd0c5c36f72b45bfc52d55094a56" exitCode=0
Jul 20 10:15:18.673657 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:18.673637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" event={"ID":"d53fb653-f539-4be4-9dd6-7934a76cc3fe","Type":"ContainerDied","Data":"2bdac40b66e47fa0579e92a9a44df2d9458ccd0c5c36f72b45bfc52d55094a56"}
Jul 20 10:15:19.250323 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:19.250277    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 20 10:15:24.249904 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:24.249858    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 20 10:15:24.256329 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:24.256289    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 20 10:15:24.256486 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:24.256429    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:15:24.256680 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:24.256652    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 10:15:24.256774 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:24.256760    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:15:27.687123 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.687083    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:15:27.697703 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.697672    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kserve-provision-location\") pod \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") "
Jul 20 10:15:27.697825 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.697781    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7b3b6fb-7783-4df3-942b-e80dbb42445c-proxy-tls\") pod \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") "
Jul 20 10:15:27.697825 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.697812    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzgp9\" (UniqueName: \"kubernetes.io/projected/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kube-api-access-gzgp9\") pod \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") "
Jul 20 10:15:27.697940 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.697844    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kube-rbac-proxy-sar-config\") pod \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") "
Jul 20 10:15:27.697940 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.697874    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7b3b6fb-7783-4df3-942b-e80dbb42445c-cabundle-cert\") pod \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\" (UID: \"a7b3b6fb-7783-4df3-942b-e80dbb42445c\") "
Jul 20 10:15:27.698044 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.697970    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7b3b6fb-7783-4df3-942b-e80dbb42445c" (UID: "a7b3b6fb-7783-4df3-942b-e80dbb42445c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:15:27.698100 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.698070    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:15:27.698388 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.698361    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b3b6fb-7783-4df3-942b-e80dbb42445c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7b3b6fb-7783-4df3-942b-e80dbb42445c" (UID: "a7b3b6fb-7783-4df3-942b-e80dbb42445c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:15:27.698515 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.698491    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7b3b6fb-7783-4df3-942b-e80dbb42445c" (UID: "a7b3b6fb-7783-4df3-942b-e80dbb42445c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:15:27.700227 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.700193    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7b3b6fb-7783-4df3-942b-e80dbb42445c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7b3b6fb-7783-4df3-942b-e80dbb42445c" (UID: "a7b3b6fb-7783-4df3-942b-e80dbb42445c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:15:27.700321 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.700302    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kube-api-access-gzgp9" (OuterVolumeSpecName: "kube-api-access-gzgp9") pod "a7b3b6fb-7783-4df3-942b-e80dbb42445c" (UID: "a7b3b6fb-7783-4df3-942b-e80dbb42445c"). InnerVolumeSpecName "kube-api-access-gzgp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:15:27.719068 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.719035    2561 generic.go:356] "Generic (PLEG): container finished" podID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerID="d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271" exitCode=0
Jul 20 10:15:27.719200 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.719104    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"
Jul 20 10:15:27.719200 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.719129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" event={"ID":"a7b3b6fb-7783-4df3-942b-e80dbb42445c","Type":"ContainerDied","Data":"d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271"}
Jul 20 10:15:27.719320 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.719203    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2" event={"ID":"a7b3b6fb-7783-4df3-942b-e80dbb42445c","Type":"ContainerDied","Data":"0603e2f0691b25348f5f1b63bc75a00766a3eacb2e3533db17a5b0153ff36d3e"}
Jul 20 10:15:27.719320 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.719251    2561 scope.go:117] "RemoveContainer" containerID="d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271"
Jul 20 10:15:27.735166 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.735141    2561 scope.go:117] "RemoveContainer" containerID="3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd"
Jul 20 10:15:27.743079 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.743058    2561 scope.go:117] "RemoveContainer" containerID="8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671"
Jul 20 10:15:27.749238 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.749210    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"]
Jul 20 10:15:27.752575 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.752554    2561 scope.go:117] "RemoveContainer" containerID="6c97cc83074f743feceeb2d8e49c5f59077d9aa35fb3fd8cee28a1e7b7dd62d7"
Jul 20 10:15:27.753015 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.752981    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5496cb9695-v2lg2"]
Jul 20 10:15:27.760413 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.760394    2561 scope.go:117] "RemoveContainer" containerID="d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271"
Jul 20 10:15:27.760708 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:15:27.760683    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271\": container with ID starting with d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271 not found: ID does not exist" containerID="d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271"
Jul 20 10:15:27.760767 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.760710    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271"} err="failed to get container status \"d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271\": rpc error: code = NotFound desc = could not find container \"d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271\": container with ID starting with d58327a57b21c0d741f570158dfd9e4389c27a46bf9cb3e340d8fd5304bfa271 not found: ID does not exist"
Jul 20 10:15:27.760767 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.760727    2561 scope.go:117] "RemoveContainer" containerID="3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd"
Jul 20 10:15:27.760953 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:15:27.760937    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd\": container with ID starting with 3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd not found: ID does not exist" containerID="3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd"
Jul 20 10:15:27.760998 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.760960    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd"} err="failed to get container status \"3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd\": rpc error: code = NotFound desc = could not find container \"3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd\": container with ID starting with 3cd5fd48ff1a87e81c9adc8b33454c7c9247b19c8f31f33e939d805455123bdd not found: ID does not exist"
Jul 20 10:15:27.760998 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.760976    2561 scope.go:117] "RemoveContainer" containerID="8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671"
Jul 20 10:15:27.761269 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:15:27.761252    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671\": container with ID starting with 8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671 not found: ID does not exist" containerID="8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671"
Jul 20 10:15:27.761354 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.761273    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671"} err="failed to get container status \"8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671\": rpc error: code = NotFound desc = could not find container \"8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671\": container with ID starting with 8fb8193c65e657fd48af607c646d4c13a432905df40650257b3f5203aca8b671 not found: ID does not exist"
Jul 20 10:15:27.761354 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.761290    2561 scope.go:117] "RemoveContainer" containerID="6c97cc83074f743feceeb2d8e49c5f59077d9aa35fb3fd8cee28a1e7b7dd62d7"
Jul 20 10:15:27.761632 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:15:27.761611    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c97cc83074f743feceeb2d8e49c5f59077d9aa35fb3fd8cee28a1e7b7dd62d7\": container with ID starting with 6c97cc83074f743feceeb2d8e49c5f59077d9aa35fb3fd8cee28a1e7b7dd62d7 not found: ID does not exist" containerID="6c97cc83074f743feceeb2d8e49c5f59077d9aa35fb3fd8cee28a1e7b7dd62d7"
Jul 20 10:15:27.761684 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.761640    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c97cc83074f743feceeb2d8e49c5f59077d9aa35fb3fd8cee28a1e7b7dd62d7"} err="failed to get container status \"6c97cc83074f743feceeb2d8e49c5f59077d9aa35fb3fd8cee28a1e7b7dd62d7\": rpc error: code = NotFound desc = could not find container \"6c97cc83074f743feceeb2d8e49c5f59077d9aa35fb3fd8cee28a1e7b7dd62d7\": container with ID starting with 6c97cc83074f743feceeb2d8e49c5f59077d9aa35fb3fd8cee28a1e7b7dd62d7 not found: ID does not exist"
Jul 20 10:15:27.799217 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.799167    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7b3b6fb-7783-4df3-942b-e80dbb42445c-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:15:27.799217 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.799191    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzgp9\" (UniqueName: \"kubernetes.io/projected/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kube-api-access-gzgp9\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:15:27.799217 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.799201    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7b3b6fb-7783-4df3-942b-e80dbb42445c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:15:27.799217 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:27.799210    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7b3b6fb-7783-4df3-942b-e80dbb42445c-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:15:28.766612 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:15:28.766579    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" path="/var/lib/kubelet/pods/a7b3b6fb-7783-4df3-942b-e80dbb42445c/volumes"
Jul 20 10:17:16.004490 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:16.004463    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:17:16.005032 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:16.004597    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:17:17.164257 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:17.164229    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" event={"ID":"d53fb653-f539-4be4-9dd6-7934a76cc3fe","Type":"ContainerStarted","Data":"25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e"}
Jul 20 10:17:17.164673 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:17.164281    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" event={"ID":"d53fb653-f539-4be4-9dd6-7934a76cc3fe","Type":"ContainerStarted","Data":"fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176"}
Jul 20 10:17:17.164673 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:17.164395    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:17:17.164673 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:17.164434    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:17:17.165669 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:17.165641    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 20 10:17:17.212044 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:17.212001    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" podStartSLOduration=1.955725942 podStartE2EDuration="2m20.211987502s" totalImagesPullingTime="1m57.894525134s" totalInitContainerRuntime="20.361736426s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:17:17.211471654 +0000 UTC m=+1217.049570239" watchObservedRunningTime="2026-07-20 10:17:17.211987502 +0000 UTC m=+1217.050086087"
Jul 20 10:17:18.168436 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:18.168394    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 20 10:17:23.173286 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:23.173252    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:17:23.173867 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:23.173840    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 20 10:17:33.174771 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:33.174742    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:17:43.669290 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:43.669256    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"]
Jul 20 10:17:43.669771 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:43.669484    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="kserve-container" containerID="cri-o://fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176" gracePeriod=30
Jul 20 10:17:43.669771 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:43.669507    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="kube-rbac-proxy" containerID="cri-o://25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e" gracePeriod=30
Jul 20 10:17:44.270078 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:44.270040    2561 generic.go:356] "Generic (PLEG): container finished" podID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerID="25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e" exitCode=2
Jul 20 10:17:44.270262 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:44.270127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" event={"ID":"d53fb653-f539-4be4-9dd6-7934a76cc3fe","Type":"ContainerDied","Data":"25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e"}
Jul 20 10:17:45.047968 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.047943    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:17:45.170216 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.170183    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kube-rbac-proxy-sar-config\") pod \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") "
Jul 20 10:17:45.170386 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.170222    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kserve-provision-location\") pod \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") "
Jul 20 10:17:45.170386 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.170260    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d53fb653-f539-4be4-9dd6-7934a76cc3fe-cabundle-cert\") pod \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") "
Jul 20 10:17:45.170386 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.170286    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmsmr\" (UniqueName: \"kubernetes.io/projected/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kube-api-access-gmsmr\") pod \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") "
Jul 20 10:17:45.170386 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.170309    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d53fb653-f539-4be4-9dd6-7934a76cc3fe-proxy-tls\") pod \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\" (UID: \"d53fb653-f539-4be4-9dd6-7934a76cc3fe\") "
Jul 20 10:17:45.170643 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.170614    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d53fb653-f539-4be4-9dd6-7934a76cc3fe" (UID: "d53fb653-f539-4be4-9dd6-7934a76cc3fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:17:45.170691 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.170621    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53fb653-f539-4be4-9dd6-7934a76cc3fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d53fb653-f539-4be4-9dd6-7934a76cc3fe" (UID: "d53fb653-f539-4be4-9dd6-7934a76cc3fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:17:45.172409 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.172376    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kube-api-access-gmsmr" (OuterVolumeSpecName: "kube-api-access-gmsmr") pod "d53fb653-f539-4be4-9dd6-7934a76cc3fe" (UID: "d53fb653-f539-4be4-9dd6-7934a76cc3fe"). InnerVolumeSpecName "kube-api-access-gmsmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:17:45.172513 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.172418    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53fb653-f539-4be4-9dd6-7934a76cc3fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d53fb653-f539-4be4-9dd6-7934a76cc3fe" (UID: "d53fb653-f539-4be4-9dd6-7934a76cc3fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:17:45.176909 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.176884    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d53fb653-f539-4be4-9dd6-7934a76cc3fe" (UID: "d53fb653-f539-4be4-9dd6-7934a76cc3fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:45.271021 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.270996    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:17:45.271021 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.271019    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:17:45.271021 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.271029    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d53fb653-f539-4be4-9dd6-7934a76cc3fe-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:17:45.271021 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.271038    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmsmr\" (UniqueName: \"kubernetes.io/projected/d53fb653-f539-4be4-9dd6-7934a76cc3fe-kube-api-access-gmsmr\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:17:45.271269 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.271048    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d53fb653-f539-4be4-9dd6-7934a76cc3fe-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:17:45.275432 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.275406    2561 generic.go:356] "Generic (PLEG): container finished" podID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerID="fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176" exitCode=0
Jul 20 10:17:45.275549 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.275469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" event={"ID":"d53fb653-f539-4be4-9dd6-7934a76cc3fe","Type":"ContainerDied","Data":"fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176"}
Jul 20 10:17:45.275549 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.275477    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"
Jul 20 10:17:45.275549 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.275516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5" event={"ID":"d53fb653-f539-4be4-9dd6-7934a76cc3fe","Type":"ContainerDied","Data":"5f160af285b0310f43200db25053350b26596c025292d7a2cf4181251aceb7e5"}
Jul 20 10:17:45.275549 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.275542    2561 scope.go:117] "RemoveContainer" containerID="25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e"
Jul 20 10:17:45.284746 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.284731    2561 scope.go:117] "RemoveContainer" containerID="fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176"
Jul 20 10:17:45.291925 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.291899    2561 scope.go:117] "RemoveContainer" containerID="2bdac40b66e47fa0579e92a9a44df2d9458ccd0c5c36f72b45bfc52d55094a56"
Jul 20 10:17:45.298719 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.298653    2561 scope.go:117] "RemoveContainer" containerID="25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e"
Jul 20 10:17:45.298935 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:17:45.298901    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e\": container with ID starting with 25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e not found: ID does not exist" containerID="25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e"
Jul 20 10:17:45.299029 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.298934    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e"} err="failed to get container status \"25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e\": rpc error: code = NotFound desc = could not find container \"25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e\": container with ID starting with 25e4ab72c2fb9b61f74ad6965b7203a6fdb641340e2b5c491fc9b7a4e2d7034e not found: ID does not exist"
Jul 20 10:17:45.299029 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.298957    2561 scope.go:117] "RemoveContainer" containerID="fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176"
Jul 20 10:17:45.299299 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:17:45.299271    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176\": container with ID starting with fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176 not found: ID does not exist" containerID="fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176"
Jul 20 10:17:45.299378 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.299305    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176"} err="failed to get container status \"fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176\": rpc error: code = NotFound desc = could not find container \"fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176\": container with ID starting with fb32f810a2c271bc5eafd64bebbc1a61b0c656341496ae5b291602ad060d0176 not found: ID does not exist"
Jul 20 10:17:45.299378 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.299328    2561 scope.go:117] "RemoveContainer" containerID="2bdac40b66e47fa0579e92a9a44df2d9458ccd0c5c36f72b45bfc52d55094a56"
Jul 20 10:17:45.299708 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:17:45.299688    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bdac40b66e47fa0579e92a9a44df2d9458ccd0c5c36f72b45bfc52d55094a56\": container with ID starting with 2bdac40b66e47fa0579e92a9a44df2d9458ccd0c5c36f72b45bfc52d55094a56 not found: ID does not exist" containerID="2bdac40b66e47fa0579e92a9a44df2d9458ccd0c5c36f72b45bfc52d55094a56"
Jul 20 10:17:45.299786 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.299716    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bdac40b66e47fa0579e92a9a44df2d9458ccd0c5c36f72b45bfc52d55094a56"} err="failed to get container status \"2bdac40b66e47fa0579e92a9a44df2d9458ccd0c5c36f72b45bfc52d55094a56\": rpc error: code = NotFound desc = could not find container \"2bdac40b66e47fa0579e92a9a44df2d9458ccd0c5c36f72b45bfc52d55094a56\": container with ID starting with 2bdac40b66e47fa0579e92a9a44df2d9458ccd0c5c36f72b45bfc52d55094a56 not found: ID does not exist"
Jul 20 10:17:45.300804 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.300785    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"]
Jul 20 10:17:45.306650 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:45.306631    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bp2w5"]
Jul 20 10:17:46.765587 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:46.765555    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" path="/var/lib/kubelet/pods/d53fb653-f539-4be4-9dd6-7934a76cc3fe/volumes"
Jul 20 10:17:48.954717 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.954688    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"]
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.954928    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.954941    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.954948    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="kserve-container"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.954953    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="kserve-container"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.954964    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb64b36-d3b0-4d82-859a-a0aca37a4690" containerName="kube-rbac-proxy"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.954969    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb64b36-d3b0-4d82-859a-a0aca37a4690" containerName="kube-rbac-proxy"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.954979    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kube-rbac-proxy"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.954984    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kube-rbac-proxy"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.954992    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb64b36-d3b0-4d82-859a-a0aca37a4690" containerName="kserve-container"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.954997    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb64b36-d3b0-4d82-859a-a0aca37a4690" containerName="kserve-container"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955002    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="storage-initializer"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955008    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="storage-initializer"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955014    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955019    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955024    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="kube-rbac-proxy"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955029    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="kube-rbac-proxy"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955035    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="storage-initializer"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955040    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="storage-initializer"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955086    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="kube-rbac-proxy"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955096    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb64b36-d3b0-4d82-859a-a0aca37a4690" containerName="kserve-container"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955102    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb64b36-d3b0-4d82-859a-a0aca37a4690" containerName="kube-rbac-proxy"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955107    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kserve-container"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955127    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="kube-rbac-proxy"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955137    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53fb653-f539-4be4-9dd6-7934a76cc3fe" containerName="kserve-container"
Jul 20 10:17:48.966890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.955144    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7b3b6fb-7783-4df3-942b-e80dbb42445c" containerName="agent"
Jul 20 10:17:48.973199 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.973183    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:48.986025 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.986007    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:17:48.986342 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.986326    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:17:48.986602 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.986588    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 20 10:17:48.986871 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.986857    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:17:48.987233 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.987216    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:17:48.987767 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.987751    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:17:48.988814 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.988800    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 20 10:17:48.997897 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:48.997875    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"]
Jul 20 10:17:49.100800 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.100778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee36b3ef-1353-4f45-9440-c10952975a02-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.100915 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.100806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee36b3ef-1353-4f45-9440-c10952975a02-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.100915 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.100844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee36b3ef-1353-4f45-9440-c10952975a02-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.100915 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.100883    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4r2q\" (UniqueName: \"kubernetes.io/projected/ee36b3ef-1353-4f45-9440-c10952975a02-kube-api-access-w4r2q\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.101040 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.100920    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee36b3ef-1353-4f45-9440-c10952975a02-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.201678 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.201644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee36b3ef-1353-4f45-9440-c10952975a02-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.201678 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.201681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4r2q\" (UniqueName: \"kubernetes.io/projected/ee36b3ef-1353-4f45-9440-c10952975a02-kube-api-access-w4r2q\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.201891 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.201701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee36b3ef-1353-4f45-9440-c10952975a02-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.201891 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.201744    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee36b3ef-1353-4f45-9440-c10952975a02-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.201891 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.201764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee36b3ef-1353-4f45-9440-c10952975a02-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.202202 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.202184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee36b3ef-1353-4f45-9440-c10952975a02-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.202351 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.202332    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee36b3ef-1353-4f45-9440-c10952975a02-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.202425 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.202407    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee36b3ef-1353-4f45-9440-c10952975a02-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.204235 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.204211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee36b3ef-1353-4f45-9440-c10952975a02-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.217714 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.217662    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4r2q\" (UniqueName: \"kubernetes.io/projected/ee36b3ef-1353-4f45-9440-c10952975a02-kube-api-access-w4r2q\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-4vgp9\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.282601 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.282571    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:17:49.434681 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:49.434595    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"]
Jul 20 10:17:49.436628 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:17:49.436599    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee36b3ef_1353_4f45_9440_c10952975a02.slice/crio-d0e65c7f63cd481bfe4857b0ad2d02e87b57f649a8bff016859371cf7dcdb93b WatchSource:0}: Error finding container d0e65c7f63cd481bfe4857b0ad2d02e87b57f649a8bff016859371cf7dcdb93b: Status 404 returned error can't find the container with id d0e65c7f63cd481bfe4857b0ad2d02e87b57f649a8bff016859371cf7dcdb93b
Jul 20 10:17:50.295405 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:50.295363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9" event={"ID":"ee36b3ef-1353-4f45-9440-c10952975a02","Type":"ContainerStarted","Data":"829d0b7f18ec33ca100233248e07a55c18b4d7d26f0e47e4914afab0fcd36381"}
Jul 20 10:17:50.295790 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:17:50.295433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9" event={"ID":"ee36b3ef-1353-4f45-9440-c10952975a02","Type":"ContainerStarted","Data":"d0e65c7f63cd481bfe4857b0ad2d02e87b57f649a8bff016859371cf7dcdb93b"}
Jul 20 10:18:20.401846 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:18:20.401816    2561 generic.go:356] "Generic (PLEG): container finished" podID="ee36b3ef-1353-4f45-9440-c10952975a02" containerID="829d0b7f18ec33ca100233248e07a55c18b4d7d26f0e47e4914afab0fcd36381" exitCode=0
Jul 20 10:18:20.402274 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:18:20.401895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9" event={"ID":"ee36b3ef-1353-4f45-9440-c10952975a02","Type":"ContainerDied","Data":"829d0b7f18ec33ca100233248e07a55c18b4d7d26f0e47e4914afab0fcd36381"}
Jul 20 10:18:21.407124 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:18:21.407089    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9" event={"ID":"ee36b3ef-1353-4f45-9440-c10952975a02","Type":"ContainerStarted","Data":"57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f"}
Jul 20 10:18:21.407491 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:18:21.407164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9" event={"ID":"ee36b3ef-1353-4f45-9440-c10952975a02","Type":"ContainerStarted","Data":"ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e"}
Jul 20 10:18:21.407491 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:18:21.407337    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:18:21.432026 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:18:21.431972    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9" podStartSLOduration=3.217487898 podStartE2EDuration="33.431954617s" totalImagesPullingTime="0s" totalInitContainerRuntime="30.214466719s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:18:21.429228509 +0000 UTC m=+1281.267327115" watchObservedRunningTime="2026-07-20 10:18:21.431954617 +0000 UTC m=+1281.270053202"
Jul 20 10:18:22.410560 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:18:22.410528    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:18:28.419817 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:18:28.419788    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:19:59.765242 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:19:59.765165    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:20:10.802319 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:10.802272    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9" podUID="ee36b3ef-1353-4f45-9440-c10952975a02" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.28:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 10:20:14.923010 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:14.922977    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"]
Jul 20 10:20:14.923454 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:14.923262    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9" podUID="ee36b3ef-1353-4f45-9440-c10952975a02" containerName="kserve-container" containerID="cri-o://ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e" gracePeriod=30
Jul 20 10:20:14.923454 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:14.923269    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9" podUID="ee36b3ef-1353-4f45-9440-c10952975a02" containerName="kube-rbac-proxy" containerID="cri-o://57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f" gracePeriod=30
Jul 20 10:20:15.829476 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:15.829445    2561 generic.go:356] "Generic (PLEG): container finished" podID="ee36b3ef-1353-4f45-9440-c10952975a02" containerID="57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f" exitCode=2
Jul 20 10:20:15.829667 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:15.829529    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9" event={"ID":"ee36b3ef-1353-4f45-9440-c10952975a02","Type":"ContainerDied","Data":"57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f"}
Jul 20 10:20:16.262489 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.262465    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:20:16.324358 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.324331    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee36b3ef-1353-4f45-9440-c10952975a02-kserve-provision-location\") pod \"ee36b3ef-1353-4f45-9440-c10952975a02\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") "
Jul 20 10:20:16.324358 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.324362    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee36b3ef-1353-4f45-9440-c10952975a02-cabundle-cert\") pod \"ee36b3ef-1353-4f45-9440-c10952975a02\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") "
Jul 20 10:20:16.324553 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.324408    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee36b3ef-1353-4f45-9440-c10952975a02-kube-rbac-proxy-sar-config\") pod \"ee36b3ef-1353-4f45-9440-c10952975a02\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") "
Jul 20 10:20:16.324553 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.324439    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee36b3ef-1353-4f45-9440-c10952975a02-proxy-tls\") pod \"ee36b3ef-1353-4f45-9440-c10952975a02\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") "
Jul 20 10:20:16.324553 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.324462    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4r2q\" (UniqueName: \"kubernetes.io/projected/ee36b3ef-1353-4f45-9440-c10952975a02-kube-api-access-w4r2q\") pod \"ee36b3ef-1353-4f45-9440-c10952975a02\" (UID: \"ee36b3ef-1353-4f45-9440-c10952975a02\") "
Jul 20 10:20:16.324793 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.324765    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee36b3ef-1353-4f45-9440-c10952975a02-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ee36b3ef-1353-4f45-9440-c10952975a02" (UID: "ee36b3ef-1353-4f45-9440-c10952975a02"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:20:16.324849 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.324786    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee36b3ef-1353-4f45-9440-c10952975a02-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ee36b3ef-1353-4f45-9440-c10952975a02" (UID: "ee36b3ef-1353-4f45-9440-c10952975a02"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:20:16.326541 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.326519    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee36b3ef-1353-4f45-9440-c10952975a02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee36b3ef-1353-4f45-9440-c10952975a02" (UID: "ee36b3ef-1353-4f45-9440-c10952975a02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:20:16.326596 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.326564    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee36b3ef-1353-4f45-9440-c10952975a02-kube-api-access-w4r2q" (OuterVolumeSpecName: "kube-api-access-w4r2q") pod "ee36b3ef-1353-4f45-9440-c10952975a02" (UID: "ee36b3ef-1353-4f45-9440-c10952975a02"). InnerVolumeSpecName "kube-api-access-w4r2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:20:16.330783 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.330756    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee36b3ef-1353-4f45-9440-c10952975a02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee36b3ef-1353-4f45-9440-c10952975a02" (UID: "ee36b3ef-1353-4f45-9440-c10952975a02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:16.425630 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.425573    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee36b3ef-1353-4f45-9440-c10952975a02-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:20:16.425630 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.425592    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee36b3ef-1353-4f45-9440-c10952975a02-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:20:16.425630 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.425603    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4r2q\" (UniqueName: \"kubernetes.io/projected/ee36b3ef-1353-4f45-9440-c10952975a02-kube-api-access-w4r2q\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:20:16.425630 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.425611    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee36b3ef-1353-4f45-9440-c10952975a02-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:20:16.425630 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.425621    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee36b3ef-1353-4f45-9440-c10952975a02-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:20:16.834869 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.834840    2561 generic.go:356] "Generic (PLEG): container finished" podID="ee36b3ef-1353-4f45-9440-c10952975a02" containerID="ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e" exitCode=0
Jul 20 10:20:16.834983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.834892    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"
Jul 20 10:20:16.834983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.834915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9" event={"ID":"ee36b3ef-1353-4f45-9440-c10952975a02","Type":"ContainerDied","Data":"ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e"}
Jul 20 10:20:16.834983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.834980    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9" event={"ID":"ee36b3ef-1353-4f45-9440-c10952975a02","Type":"ContainerDied","Data":"d0e65c7f63cd481bfe4857b0ad2d02e87b57f649a8bff016859371cf7dcdb93b"}
Jul 20 10:20:16.835088 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.835016    2561 scope.go:117] "RemoveContainer" containerID="57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f"
Jul 20 10:20:16.844926 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.844909    2561 scope.go:117] "RemoveContainer" containerID="ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e"
Jul 20 10:20:16.851734 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.851719    2561 scope.go:117] "RemoveContainer" containerID="829d0b7f18ec33ca100233248e07a55c18b4d7d26f0e47e4914afab0fcd36381"
Jul 20 10:20:16.855459 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.855435    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"]
Jul 20 10:20:16.859584 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.859563    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-4vgp9"]
Jul 20 10:20:16.860390 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.860378    2561 scope.go:117] "RemoveContainer" containerID="57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f"
Jul 20 10:20:16.860640 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:20:16.860624    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f\": container with ID starting with 57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f not found: ID does not exist" containerID="57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f"
Jul 20 10:20:16.860683 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.860646    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f"} err="failed to get container status \"57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f\": rpc error: code = NotFound desc = could not find container \"57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f\": container with ID starting with 57e8ded917163479f2f716da09abf63afc94e5188a07d35caf6ba1615598795f not found: ID does not exist"
Jul 20 10:20:16.860683 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.860662    2561 scope.go:117] "RemoveContainer" containerID="ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e"
Jul 20 10:20:16.860895 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:20:16.860876    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e\": container with ID starting with ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e not found: ID does not exist" containerID="ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e"
Jul 20 10:20:16.860950 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.860899    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e"} err="failed to get container status \"ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e\": rpc error: code = NotFound desc = could not find container \"ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e\": container with ID starting with ad0cf282ee8e1f77a70fbdb5c8d7a71c87a1915529125dc259671667a80b522e not found: ID does not exist"
Jul 20 10:20:16.860950 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.860913    2561 scope.go:117] "RemoveContainer" containerID="829d0b7f18ec33ca100233248e07a55c18b4d7d26f0e47e4914afab0fcd36381"
Jul 20 10:20:16.861091 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:20:16.861077    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829d0b7f18ec33ca100233248e07a55c18b4d7d26f0e47e4914afab0fcd36381\": container with ID starting with 829d0b7f18ec33ca100233248e07a55c18b4d7d26f0e47e4914afab0fcd36381 not found: ID does not exist" containerID="829d0b7f18ec33ca100233248e07a55c18b4d7d26f0e47e4914afab0fcd36381"
Jul 20 10:20:16.861151 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:16.861095    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829d0b7f18ec33ca100233248e07a55c18b4d7d26f0e47e4914afab0fcd36381"} err="failed to get container status \"829d0b7f18ec33ca100233248e07a55c18b4d7d26f0e47e4914afab0fcd36381\": rpc error: code = NotFound desc = could not find container \"829d0b7f18ec33ca100233248e07a55c18b4d7d26f0e47e4914afab0fcd36381\": container with ID starting with 829d0b7f18ec33ca100233248e07a55c18b4d7d26f0e47e4914afab0fcd36381 not found: ID does not exist"
Jul 20 10:20:18.765567 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:18.765534    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee36b3ef-1353-4f45-9440-c10952975a02" path="/var/lib/kubelet/pods/ee36b3ef-1353-4f45-9440-c10952975a02/volumes"
Jul 20 10:20:20.167286 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.167258    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"]
Jul 20 10:20:20.167650 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.167496    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee36b3ef-1353-4f45-9440-c10952975a02" containerName="storage-initializer"
Jul 20 10:20:20.167650 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.167507    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee36b3ef-1353-4f45-9440-c10952975a02" containerName="storage-initializer"
Jul 20 10:20:20.167650 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.167522    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee36b3ef-1353-4f45-9440-c10952975a02" containerName="kserve-container"
Jul 20 10:20:20.167650 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.167527    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee36b3ef-1353-4f45-9440-c10952975a02" containerName="kserve-container"
Jul 20 10:20:20.167650 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.167535    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee36b3ef-1353-4f45-9440-c10952975a02" containerName="kube-rbac-proxy"
Jul 20 10:20:20.167650 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.167541    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee36b3ef-1353-4f45-9440-c10952975a02" containerName="kube-rbac-proxy"
Jul 20 10:20:20.167650 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.167583    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee36b3ef-1353-4f45-9440-c10952975a02" containerName="kube-rbac-proxy"
Jul 20 10:20:20.167650 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.167591    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee36b3ef-1353-4f45-9440-c10952975a02" containerName="kserve-container"
Jul 20 10:20:20.170635 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.170615    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.174640 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.174618    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:20:20.174745 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.174618    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:20:20.174745 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.174703    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:20:20.175361 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.175334    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:20:20.175448 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.175416    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 20 10:20:20.175800 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.175786    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 20 10:20:20.175860 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.175823    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:20:20.187594 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.187565    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"]
Jul 20 10:20:20.249331 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.249299    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d172ec6a-c861-4413-9b31-980d622f37cc-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.249468 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.249342    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d172ec6a-c861-4413-9b31-980d622f37cc-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.249468 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.249390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d172ec6a-c861-4413-9b31-980d622f37cc-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.249468 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.249420    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfqq\" (UniqueName: \"kubernetes.io/projected/d172ec6a-c861-4413-9b31-980d622f37cc-kube-api-access-tzfqq\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.249611 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.249473    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d172ec6a-c861-4413-9b31-980d622f37cc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.350455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.350423    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d172ec6a-c861-4413-9b31-980d622f37cc-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.350620 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.350463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfqq\" (UniqueName: \"kubernetes.io/projected/d172ec6a-c861-4413-9b31-980d622f37cc-kube-api-access-tzfqq\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.350620 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.350499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d172ec6a-c861-4413-9b31-980d622f37cc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.350759 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:20:20.350613    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 20 10:20:20.350759 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:20:20.350687    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d172ec6a-c861-4413-9b31-980d622f37cc-proxy-tls podName:d172ec6a-c861-4413-9b31-980d622f37cc nodeName:}" failed. No retries permitted until 2026-07-20 10:20:20.850665261 +0000 UTC m=+1400.688763839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d172ec6a-c861-4413-9b31-980d622f37cc-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-thq92" (UID: "d172ec6a-c861-4413-9b31-980d622f37cc") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 20 10:20:20.350759 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.350620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d172ec6a-c861-4413-9b31-980d622f37cc-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.350950 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.350765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d172ec6a-c861-4413-9b31-980d622f37cc-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.350988 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.350942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d172ec6a-c861-4413-9b31-980d622f37cc-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.351226 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.351209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d172ec6a-c861-4413-9b31-980d622f37cc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.351330 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.351314    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d172ec6a-c861-4413-9b31-980d622f37cc-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.361516 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.361483    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfqq\" (UniqueName: \"kubernetes.io/projected/d172ec6a-c861-4413-9b31-980d622f37cc-kube-api-access-tzfqq\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.854956 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.854916    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d172ec6a-c861-4413-9b31-980d622f37cc-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:20.857202 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:20.857183    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d172ec6a-c861-4413-9b31-980d622f37cc-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-thq92\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:21.079939 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:21.079905    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:21.237415 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:21.237382    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"]
Jul 20 10:20:21.240733 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:20:21.240706    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd172ec6a_c861_4413_9b31_980d622f37cc.slice/crio-887afbd281330eff1c7fc24b01c9887c7f5c856b4cd59f40e6f59fd9d9cc9396 WatchSource:0}: Error finding container 887afbd281330eff1c7fc24b01c9887c7f5c856b4cd59f40e6f59fd9d9cc9396: Status 404 returned error can't find the container with id 887afbd281330eff1c7fc24b01c9887c7f5c856b4cd59f40e6f59fd9d9cc9396
Jul 20 10:20:21.853992 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:21.853953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92" event={"ID":"d172ec6a-c861-4413-9b31-980d622f37cc","Type":"ContainerStarted","Data":"ee176115d2e464206eab8902a0066c9eaf531869e27460246e3692e8011fd15a"}
Jul 20 10:20:21.854220 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:21.854009    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92" event={"ID":"d172ec6a-c861-4413-9b31-980d622f37cc","Type":"ContainerStarted","Data":"887afbd281330eff1c7fc24b01c9887c7f5c856b4cd59f40e6f59fd9d9cc9396"}
Jul 20 10:20:25.875394 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:25.875361    2561 generic.go:356] "Generic (PLEG): container finished" podID="d172ec6a-c861-4413-9b31-980d622f37cc" containerID="ee176115d2e464206eab8902a0066c9eaf531869e27460246e3692e8011fd15a" exitCode=0
Jul 20 10:20:25.875728 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:25.875423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92" event={"ID":"d172ec6a-c861-4413-9b31-980d622f37cc","Type":"ContainerDied","Data":"ee176115d2e464206eab8902a0066c9eaf531869e27460246e3692e8011fd15a"}
Jul 20 10:20:26.884346 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:26.884315    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92" event={"ID":"d172ec6a-c861-4413-9b31-980d622f37cc","Type":"ContainerStarted","Data":"87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240"}
Jul 20 10:20:26.884736 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:26.884381    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92" event={"ID":"d172ec6a-c861-4413-9b31-980d622f37cc","Type":"ContainerStarted","Data":"f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e"}
Jul 20 10:20:26.884736 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:26.884578    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:26.906270 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:26.906225    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92" podStartSLOduration=2.469177706 podStartE2EDuration="6.906210455s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.437032749s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:20:26.904954129 +0000 UTC m=+1406.743052744" watchObservedRunningTime="2026-07-20 10:20:26.906210455 +0000 UTC m=+1406.744309040"
Jul 20 10:20:27.889516 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:27.889478    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:20:33.898101 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:20:33.898069    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:21:55.765569 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:21:55.765529    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:22:06.623882 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:06.623848    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"]
Jul 20 10:22:06.624345 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:06.624150    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92" podUID="d172ec6a-c861-4413-9b31-980d622f37cc" containerName="kserve-container" containerID="cri-o://f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e" gracePeriod=30
Jul 20 10:22:06.624345 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:06.624160    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92" podUID="d172ec6a-c861-4413-9b31-980d622f37cc" containerName="kube-rbac-proxy" containerID="cri-o://87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240" gracePeriod=30
Jul 20 10:22:07.259564 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.259528    2561 generic.go:356] "Generic (PLEG): container finished" podID="d172ec6a-c861-4413-9b31-980d622f37cc" containerID="87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240" exitCode=2
Jul 20 10:22:07.259750 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.259602    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92" event={"ID":"d172ec6a-c861-4413-9b31-980d622f37cc","Type":"ContainerDied","Data":"87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240"}
Jul 20 10:22:07.861748 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.861724    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:22:07.920142 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.920105    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d172ec6a-c861-4413-9b31-980d622f37cc-proxy-tls\") pod \"d172ec6a-c861-4413-9b31-980d622f37cc\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") "
Jul 20 10:22:07.920279 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.920164    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d172ec6a-c861-4413-9b31-980d622f37cc-kube-rbac-proxy-sar-config\") pod \"d172ec6a-c861-4413-9b31-980d622f37cc\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") "
Jul 20 10:22:07.920279 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.920202    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d172ec6a-c861-4413-9b31-980d622f37cc-cabundle-cert\") pod \"d172ec6a-c861-4413-9b31-980d622f37cc\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") "
Jul 20 10:22:07.920279 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.920231    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d172ec6a-c861-4413-9b31-980d622f37cc-kserve-provision-location\") pod \"d172ec6a-c861-4413-9b31-980d622f37cc\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") "
Jul 20 10:22:07.920279 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.920258    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzfqq\" (UniqueName: \"kubernetes.io/projected/d172ec6a-c861-4413-9b31-980d622f37cc-kube-api-access-tzfqq\") pod \"d172ec6a-c861-4413-9b31-980d622f37cc\" (UID: \"d172ec6a-c861-4413-9b31-980d622f37cc\") "
Jul 20 10:22:07.920584 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.920560    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d172ec6a-c861-4413-9b31-980d622f37cc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d172ec6a-c861-4413-9b31-980d622f37cc" (UID: "d172ec6a-c861-4413-9b31-980d622f37cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:22:07.920648 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.920557    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d172ec6a-c861-4413-9b31-980d622f37cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d172ec6a-c861-4413-9b31-980d622f37cc" (UID: "d172ec6a-c861-4413-9b31-980d622f37cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:22:07.922080 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.922052    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d172ec6a-c861-4413-9b31-980d622f37cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d172ec6a-c861-4413-9b31-980d622f37cc" (UID: "d172ec6a-c861-4413-9b31-980d622f37cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:22:07.922341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.922315    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d172ec6a-c861-4413-9b31-980d622f37cc-kube-api-access-tzfqq" (OuterVolumeSpecName: "kube-api-access-tzfqq") pod "d172ec6a-c861-4413-9b31-980d622f37cc" (UID: "d172ec6a-c861-4413-9b31-980d622f37cc"). InnerVolumeSpecName "kube-api-access-tzfqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:22:07.926611 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:07.926552    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d172ec6a-c861-4413-9b31-980d622f37cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d172ec6a-c861-4413-9b31-980d622f37cc" (UID: "d172ec6a-c861-4413-9b31-980d622f37cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:22:08.021073 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.021045    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d172ec6a-c861-4413-9b31-980d622f37cc-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:22:08.021073 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.021069    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d172ec6a-c861-4413-9b31-980d622f37cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:22:08.021236 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.021079    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d172ec6a-c861-4413-9b31-980d622f37cc-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:22:08.021236 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.021088    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d172ec6a-c861-4413-9b31-980d622f37cc-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:22:08.021236 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.021097    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzfqq\" (UniqueName: \"kubernetes.io/projected/d172ec6a-c861-4413-9b31-980d622f37cc-kube-api-access-tzfqq\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:22:08.264754 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.264660    2561 generic.go:356] "Generic (PLEG): container finished" podID="d172ec6a-c861-4413-9b31-980d622f37cc" containerID="f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e" exitCode=0
Jul 20 10:22:08.264754 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.264719    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"
Jul 20 10:22:08.264967 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.264748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92" event={"ID":"d172ec6a-c861-4413-9b31-980d622f37cc","Type":"ContainerDied","Data":"f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e"}
Jul 20 10:22:08.264967 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.264825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92" event={"ID":"d172ec6a-c861-4413-9b31-980d622f37cc","Type":"ContainerDied","Data":"887afbd281330eff1c7fc24b01c9887c7f5c856b4cd59f40e6f59fd9d9cc9396"}
Jul 20 10:22:08.264967 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.264864    2561 scope.go:117] "RemoveContainer" containerID="87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240"
Jul 20 10:22:08.272953 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.272822    2561 scope.go:117] "RemoveContainer" containerID="f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e"
Jul 20 10:22:08.279885 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.279867    2561 scope.go:117] "RemoveContainer" containerID="ee176115d2e464206eab8902a0066c9eaf531869e27460246e3692e8011fd15a"
Jul 20 10:22:08.287140 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.287100    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"]
Jul 20 10:22:08.287337 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.287320    2561 scope.go:117] "RemoveContainer" containerID="87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240"
Jul 20 10:22:08.287607 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:22:08.287589    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240\": container with ID starting with 87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240 not found: ID does not exist" containerID="87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240"
Jul 20 10:22:08.287659 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.287615    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240"} err="failed to get container status \"87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240\": rpc error: code = NotFound desc = could not find container \"87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240\": container with ID starting with 87e79ceead4de93b2043f080631ce04c2f7568a92d2b16c39ad5e212d518a240 not found: ID does not exist"
Jul 20 10:22:08.287659 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.287634    2561 scope.go:117] "RemoveContainer" containerID="f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e"
Jul 20 10:22:08.287854 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:22:08.287836    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e\": container with ID starting with f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e not found: ID does not exist" containerID="f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e"
Jul 20 10:22:08.287900 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.287861    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e"} err="failed to get container status \"f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e\": rpc error: code = NotFound desc = could not find container \"f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e\": container with ID starting with f7a555889a2f28e05cf8171bfb1b0712cb7318a1c72027d8a0f2df89f3a25f3e not found: ID does not exist"
Jul 20 10:22:08.287900 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.287877    2561 scope.go:117] "RemoveContainer" containerID="ee176115d2e464206eab8902a0066c9eaf531869e27460246e3692e8011fd15a"
Jul 20 10:22:08.288072 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:22:08.288056    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee176115d2e464206eab8902a0066c9eaf531869e27460246e3692e8011fd15a\": container with ID starting with ee176115d2e464206eab8902a0066c9eaf531869e27460246e3692e8011fd15a not found: ID does not exist" containerID="ee176115d2e464206eab8902a0066c9eaf531869e27460246e3692e8011fd15a"
Jul 20 10:22:08.288144 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.288079    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee176115d2e464206eab8902a0066c9eaf531869e27460246e3692e8011fd15a"} err="failed to get container status \"ee176115d2e464206eab8902a0066c9eaf531869e27460246e3692e8011fd15a\": rpc error: code = NotFound desc = could not find container \"ee176115d2e464206eab8902a0066c9eaf531869e27460246e3692e8011fd15a\": container with ID starting with ee176115d2e464206eab8902a0066c9eaf531869e27460246e3692e8011fd15a not found: ID does not exist"
Jul 20 10:22:08.291316 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.291296    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-thq92"]
Jul 20 10:22:08.764846 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:08.764803    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d172ec6a-c861-4413-9b31-980d622f37cc" path="/var/lib/kubelet/pods/d172ec6a-c861-4413-9b31-980d622f37cc/volumes"
Jul 20 10:22:11.881950 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.881917    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"]
Jul 20 10:22:11.882334 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.882175    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d172ec6a-c861-4413-9b31-980d622f37cc" containerName="kserve-container"
Jul 20 10:22:11.882334 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.882186    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d172ec6a-c861-4413-9b31-980d622f37cc" containerName="kserve-container"
Jul 20 10:22:11.882334 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.882201    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d172ec6a-c861-4413-9b31-980d622f37cc" containerName="storage-initializer"
Jul 20 10:22:11.882334 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.882206    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d172ec6a-c861-4413-9b31-980d622f37cc" containerName="storage-initializer"
Jul 20 10:22:11.882334 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.882215    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d172ec6a-c861-4413-9b31-980d622f37cc" containerName="kube-rbac-proxy"
Jul 20 10:22:11.882334 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.882220    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d172ec6a-c861-4413-9b31-980d622f37cc" containerName="kube-rbac-proxy"
Jul 20 10:22:11.882334 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.882261    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d172ec6a-c861-4413-9b31-980d622f37cc" containerName="kserve-container"
Jul 20 10:22:11.882334 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.882268    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d172ec6a-c861-4413-9b31-980d622f37cc" containerName="kube-rbac-proxy"
Jul 20 10:22:11.886740 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.886720    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:11.889299 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.889276    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:22:11.889404 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.889288    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:22:11.889857 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.889833    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 20 10:22:11.889857 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.889839    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:22:11.890029 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.889840    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 20 10:22:11.890029 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.889972    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:22:11.890415 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.890399    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:22:11.897368 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.897349    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"]
Jul 20 10:22:11.950037 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.950012    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89caa13f-cb86-4d57-b68b-559ae2a98fef-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:11.950174 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.950043    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms9jk\" (UniqueName: \"kubernetes.io/projected/89caa13f-cb86-4d57-b68b-559ae2a98fef-kube-api-access-ms9jk\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:11.950174 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.950062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89caa13f-cb86-4d57-b68b-559ae2a98fef-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:11.950174 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.950153    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89caa13f-cb86-4d57-b68b-559ae2a98fef-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:11.950289 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:11.950198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89caa13f-cb86-4d57-b68b-559ae2a98fef-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:12.051499 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:12.051470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms9jk\" (UniqueName: \"kubernetes.io/projected/89caa13f-cb86-4d57-b68b-559ae2a98fef-kube-api-access-ms9jk\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:12.051499 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:12.051502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89caa13f-cb86-4d57-b68b-559ae2a98fef-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:12.051667 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:12.051533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89caa13f-cb86-4d57-b68b-559ae2a98fef-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:12.051745 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:12.051722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89caa13f-cb86-4d57-b68b-559ae2a98fef-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:12.051818 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:12.051805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89caa13f-cb86-4d57-b68b-559ae2a98fef-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:12.052100 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:12.052072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89caa13f-cb86-4d57-b68b-559ae2a98fef-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:12.052231 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:12.052211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89caa13f-cb86-4d57-b68b-559ae2a98fef-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:12.052334 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:12.052316    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89caa13f-cb86-4d57-b68b-559ae2a98fef-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:12.053912 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:12.053896    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89caa13f-cb86-4d57-b68b-559ae2a98fef-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:12.060243 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:12.060216    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms9jk\" (UniqueName: \"kubernetes.io/projected/89caa13f-cb86-4d57-b68b-559ae2a98fef-kube-api-access-ms9jk\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:12.197805 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:12.197727    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:12.327068 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:12.327043    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"]
Jul 20 10:22:12.329286 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:22:12.329255    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89caa13f_cb86_4d57_b68b_559ae2a98fef.slice/crio-97df5e9bcef73f07b58f3ae6829d3ce3efc759779647d586ac2af94a734c132e WatchSource:0}: Error finding container 97df5e9bcef73f07b58f3ae6829d3ce3efc759779647d586ac2af94a734c132e: Status 404 returned error can't find the container with id 97df5e9bcef73f07b58f3ae6829d3ce3efc759779647d586ac2af94a734c132e
Jul 20 10:22:13.291532 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:13.291499    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss" event={"ID":"89caa13f-cb86-4d57-b68b-559ae2a98fef","Type":"ContainerStarted","Data":"1af5a7b62b85b7b99414d62d1d50266d1f94d2c696badb9463d3f950d8ac017a"}
Jul 20 10:22:13.291897 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:13.291555    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss" event={"ID":"89caa13f-cb86-4d57-b68b-559ae2a98fef","Type":"ContainerStarted","Data":"97df5e9bcef73f07b58f3ae6829d3ce3efc759779647d586ac2af94a734c132e"}
Jul 20 10:22:16.031515 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:16.031488    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:22:16.032030 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:16.032012    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:22:42.396162 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:42.396129    2561 generic.go:356] "Generic (PLEG): container finished" podID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerID="1af5a7b62b85b7b99414d62d1d50266d1f94d2c696badb9463d3f950d8ac017a" exitCode=0
Jul 20 10:22:42.396559 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:42.396186    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss" event={"ID":"89caa13f-cb86-4d57-b68b-559ae2a98fef","Type":"ContainerDied","Data":"1af5a7b62b85b7b99414d62d1d50266d1f94d2c696badb9463d3f950d8ac017a"}
Jul 20 10:22:43.401856 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:43.401822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss" event={"ID":"89caa13f-cb86-4d57-b68b-559ae2a98fef","Type":"ContainerStarted","Data":"e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3"}
Jul 20 10:22:43.402269 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:43.401885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss" event={"ID":"89caa13f-cb86-4d57-b68b-559ae2a98fef","Type":"ContainerStarted","Data":"907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458"}
Jul 20 10:22:43.402269 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:43.402071    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:43.425738 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:43.425677    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss" podStartSLOduration=2.950343758 podStartE2EDuration="32.425655295s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.475311537s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:22:43.423566749 +0000 UTC m=+1543.261665399" watchObservedRunningTime="2026-07-20 10:22:43.425655295 +0000 UTC m=+1543.263753881"
Jul 20 10:22:44.406132 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:44.406091    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:22:50.415190 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:22:50.415157    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:24:21.764969 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:21.764935    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:24:37.929295 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:37.929264    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"]
Jul 20 10:24:37.929699 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:37.929496    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss" podUID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerName="kserve-container" containerID="cri-o://907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458" gracePeriod=30
Jul 20 10:24:37.929699 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:37.929621    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss" podUID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerName="kube-rbac-proxy" containerID="cri-o://e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3" gracePeriod=30
Jul 20 10:24:38.808444 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:38.808408    2561 generic.go:356] "Generic (PLEG): container finished" podID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerID="e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3" exitCode=2
Jul 20 10:24:38.808625 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:38.808488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss" event={"ID":"89caa13f-cb86-4d57-b68b-559ae2a98fef","Type":"ContainerDied","Data":"e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3"}
Jul 20 10:24:39.168702 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.168679    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:24:39.308104 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.308076    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89caa13f-cb86-4d57-b68b-559ae2a98fef-cabundle-cert\") pod \"89caa13f-cb86-4d57-b68b-559ae2a98fef\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") "
Jul 20 10:24:39.308299 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.308107    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms9jk\" (UniqueName: \"kubernetes.io/projected/89caa13f-cb86-4d57-b68b-559ae2a98fef-kube-api-access-ms9jk\") pod \"89caa13f-cb86-4d57-b68b-559ae2a98fef\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") "
Jul 20 10:24:39.308299 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.308180    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89caa13f-cb86-4d57-b68b-559ae2a98fef-proxy-tls\") pod \"89caa13f-cb86-4d57-b68b-559ae2a98fef\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") "
Jul 20 10:24:39.308299 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.308198    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89caa13f-cb86-4d57-b68b-559ae2a98fef-kserve-provision-location\") pod \"89caa13f-cb86-4d57-b68b-559ae2a98fef\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") "
Jul 20 10:24:39.308299 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.308228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89caa13f-cb86-4d57-b68b-559ae2a98fef-kube-rbac-proxy-sar-config\") pod \"89caa13f-cb86-4d57-b68b-559ae2a98fef\" (UID: \"89caa13f-cb86-4d57-b68b-559ae2a98fef\") "
Jul 20 10:24:39.308522 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.308441    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89caa13f-cb86-4d57-b68b-559ae2a98fef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89caa13f-cb86-4d57-b68b-559ae2a98fef" (UID: "89caa13f-cb86-4d57-b68b-559ae2a98fef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:24:39.308612 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.308587    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89caa13f-cb86-4d57-b68b-559ae2a98fef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89caa13f-cb86-4d57-b68b-559ae2a98fef" (UID: "89caa13f-cb86-4d57-b68b-559ae2a98fef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:24:39.310282 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.310259    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89caa13f-cb86-4d57-b68b-559ae2a98fef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89caa13f-cb86-4d57-b68b-559ae2a98fef" (UID: "89caa13f-cb86-4d57-b68b-559ae2a98fef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:24:39.310385 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.310260    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89caa13f-cb86-4d57-b68b-559ae2a98fef-kube-api-access-ms9jk" (OuterVolumeSpecName: "kube-api-access-ms9jk") pod "89caa13f-cb86-4d57-b68b-559ae2a98fef" (UID: "89caa13f-cb86-4d57-b68b-559ae2a98fef"). InnerVolumeSpecName "kube-api-access-ms9jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:24:39.312899 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.312866    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89caa13f-cb86-4d57-b68b-559ae2a98fef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89caa13f-cb86-4d57-b68b-559ae2a98fef" (UID: "89caa13f-cb86-4d57-b68b-559ae2a98fef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:24:39.408798 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.408764    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89caa13f-cb86-4d57-b68b-559ae2a98fef-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:24:39.408798 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.408800    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89caa13f-cb86-4d57-b68b-559ae2a98fef-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:24:39.408926 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.408810    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89caa13f-cb86-4d57-b68b-559ae2a98fef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:24:39.408926 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.408818    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89caa13f-cb86-4d57-b68b-559ae2a98fef-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:24:39.408926 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.408828    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms9jk\" (UniqueName: \"kubernetes.io/projected/89caa13f-cb86-4d57-b68b-559ae2a98fef-kube-api-access-ms9jk\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:24:39.814025 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.813944    2561 generic.go:356] "Generic (PLEG): container finished" podID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerID="907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458" exitCode=0
Jul 20 10:24:39.814025 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.814002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss" event={"ID":"89caa13f-cb86-4d57-b68b-559ae2a98fef","Type":"ContainerDied","Data":"907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458"}
Jul 20 10:24:39.814212 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.814050    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss" event={"ID":"89caa13f-cb86-4d57-b68b-559ae2a98fef","Type":"ContainerDied","Data":"97df5e9bcef73f07b58f3ae6829d3ce3efc759779647d586ac2af94a734c132e"}
Jul 20 10:24:39.814212 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.814078    2561 scope.go:117] "RemoveContainer" containerID="e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3"
Jul 20 10:24:39.814212 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.814008    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"
Jul 20 10:24:39.823852 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.823730    2561 scope.go:117] "RemoveContainer" containerID="907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458"
Jul 20 10:24:39.833773 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.833750    2561 scope.go:117] "RemoveContainer" containerID="1af5a7b62b85b7b99414d62d1d50266d1f94d2c696badb9463d3f950d8ac017a"
Jul 20 10:24:39.840496 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.840468    2561 scope.go:117] "RemoveContainer" containerID="e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3"
Jul 20 10:24:39.840835 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:24:39.840799    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3\": container with ID starting with e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3 not found: ID does not exist" containerID="e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3"
Jul 20 10:24:39.840897 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.840850    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3"} err="failed to get container status \"e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3\": rpc error: code = NotFound desc = could not find container \"e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3\": container with ID starting with e4e8970613d10ed21ba96a16206e5c6321e36f0eee003ac645f3e63835096de3 not found: ID does not exist"
Jul 20 10:24:39.840897 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.840890    2561 scope.go:117] "RemoveContainer" containerID="907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458"
Jul 20 10:24:39.841208 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:24:39.841179    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458\": container with ID starting with 907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458 not found: ID does not exist" containerID="907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458"
Jul 20 10:24:39.841288 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.841226    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458"} err="failed to get container status \"907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458\": rpc error: code = NotFound desc = could not find container \"907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458\": container with ID starting with 907477e76933f05f519978ed0c6b87eb47edf7b53d7accc989afbf4baf7f2458 not found: ID does not exist"
Jul 20 10:24:39.841288 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.841247    2561 scope.go:117] "RemoveContainer" containerID="1af5a7b62b85b7b99414d62d1d50266d1f94d2c696badb9463d3f950d8ac017a"
Jul 20 10:24:39.841539 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:24:39.841517    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af5a7b62b85b7b99414d62d1d50266d1f94d2c696badb9463d3f950d8ac017a\": container with ID starting with 1af5a7b62b85b7b99414d62d1d50266d1f94d2c696badb9463d3f950d8ac017a not found: ID does not exist" containerID="1af5a7b62b85b7b99414d62d1d50266d1f94d2c696badb9463d3f950d8ac017a"
Jul 20 10:24:39.841608 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.841545    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af5a7b62b85b7b99414d62d1d50266d1f94d2c696badb9463d3f950d8ac017a"} err="failed to get container status \"1af5a7b62b85b7b99414d62d1d50266d1f94d2c696badb9463d3f950d8ac017a\": rpc error: code = NotFound desc = could not find container \"1af5a7b62b85b7b99414d62d1d50266d1f94d2c696badb9463d3f950d8ac017a\": container with ID starting with 1af5a7b62b85b7b99414d62d1d50266d1f94d2c696badb9463d3f950d8ac017a not found: ID does not exist"
Jul 20 10:24:39.843667 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.843646    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"]
Jul 20 10:24:39.847601 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:39.847583    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-dqxss"]
Jul 20 10:24:40.766403 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:40.766362    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89caa13f-cb86-4d57-b68b-559ae2a98fef" path="/var/lib/kubelet/pods/89caa13f-cb86-4d57-b68b-559ae2a98fef/volumes"
Jul 20 10:24:43.062091 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.062061    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"]
Jul 20 10:24:43.062455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.062320    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerName="storage-initializer"
Jul 20 10:24:43.062455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.062332    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerName="storage-initializer"
Jul 20 10:24:43.062455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.062344    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerName="kserve-container"
Jul 20 10:24:43.062455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.062349    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerName="kserve-container"
Jul 20 10:24:43.062455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.062359    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerName="kube-rbac-proxy"
Jul 20 10:24:43.062455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.062367    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerName="kube-rbac-proxy"
Jul 20 10:24:43.062455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.062410    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerName="kube-rbac-proxy"
Jul 20 10:24:43.062455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.062419    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="89caa13f-cb86-4d57-b68b-559ae2a98fef" containerName="kserve-container"
Jul 20 10:24:43.066731 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.066712    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.069284 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.069263    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 20 10:24:43.069397 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.069287    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:24:43.069397 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.069299    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 20 10:24:43.069397 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.069316    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:24:43.069559 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.069411    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:24:43.069559 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.069413    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:24:43.069704 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.069689    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:24:43.076692 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.076669    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"]
Jul 20 10:24:43.238432 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.238395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.238432 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.238430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whhf6\" (UniqueName: \"kubernetes.io/projected/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kube-api-access-whhf6\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.238659 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.238452    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.238659 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.238549    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.238659 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.238622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.339773 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.339689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.339773 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.339730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whhf6\" (UniqueName: \"kubernetes.io/projected/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kube-api-access-whhf6\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.339773 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.339759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.340025 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.339794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.340025 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.339842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.340254 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.340233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.340548 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.340529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.340621 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.340529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.342061 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.342043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.348310 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.348291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whhf6\" (UniqueName: \"kubernetes.io/projected/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kube-api-access-whhf6\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.376958 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.376934    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:43.499880 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.499855    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"]
Jul 20 10:24:43.502360 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:24:43.502335    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad87b9b_9ae7_43ef_8479_a1c3447d484e.slice/crio-7cd4cd5505f75f7f6ca6da3e646c83be92535dbed48c134c088d8bf58ce19f01 WatchSource:0}: Error finding container 7cd4cd5505f75f7f6ca6da3e646c83be92535dbed48c134c088d8bf58ce19f01: Status 404 returned error can't find the container with id 7cd4cd5505f75f7f6ca6da3e646c83be92535dbed48c134c088d8bf58ce19f01
Jul 20 10:24:43.832414 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.832379    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" event={"ID":"2ad87b9b-9ae7-43ef-8479-a1c3447d484e","Type":"ContainerStarted","Data":"42ab5d3925bfb54c206370b51a29126abac4a2c4ec75761b1456b34714f3032d"}
Jul 20 10:24:43.832586 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:43.832434    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" event={"ID":"2ad87b9b-9ae7-43ef-8479-a1c3447d484e","Type":"ContainerStarted","Data":"7cd4cd5505f75f7f6ca6da3e646c83be92535dbed48c134c088d8bf58ce19f01"}
Jul 20 10:24:51.862844 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:51.862814    2561 generic.go:356] "Generic (PLEG): container finished" podID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerID="42ab5d3925bfb54c206370b51a29126abac4a2c4ec75761b1456b34714f3032d" exitCode=0
Jul 20 10:24:51.863236 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:51.862863    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" event={"ID":"2ad87b9b-9ae7-43ef-8479-a1c3447d484e","Type":"ContainerDied","Data":"42ab5d3925bfb54c206370b51a29126abac4a2c4ec75761b1456b34714f3032d"}
Jul 20 10:24:52.868775 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:52.868741    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" event={"ID":"2ad87b9b-9ae7-43ef-8479-a1c3447d484e","Type":"ContainerStarted","Data":"5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5"}
Jul 20 10:24:52.869260 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:52.868809    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" event={"ID":"2ad87b9b-9ae7-43ef-8479-a1c3447d484e","Type":"ContainerStarted","Data":"3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1"}
Jul 20 10:24:52.869260 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:52.869138    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:52.890474 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:52.890419    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" podStartSLOduration=2.041718748 podStartE2EDuration="9.890401812s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.848683064s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:24:52.889312923 +0000 UTC m=+1672.727411537" watchObservedRunningTime="2026-07-20 10:24:52.890401812 +0000 UTC m=+1672.728500397"
Jul 20 10:24:53.872889 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:53.872858    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:53.874339 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:53.874304    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 20 10:24:54.876596 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:54.876560    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 20 10:24:59.881065 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:59.881036    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:24:59.881568 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:24:59.881542    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 20 10:25:09.883066 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:09.883040    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:25:25.181059 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:25.180973    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"]
Jul 20 10:25:25.181600 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:25.181243    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="kserve-container" containerID="cri-o://3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1" gracePeriod=30
Jul 20 10:25:25.181600 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:25.181305    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="kube-rbac-proxy" containerID="cri-o://5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5" gracePeriod=30
Jul 20 10:25:25.995043 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:25.995005    2561 generic.go:356] "Generic (PLEG): container finished" podID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerID="5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5" exitCode=2
Jul 20 10:25:25.995230 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:25.995079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" event={"ID":"2ad87b9b-9ae7-43ef-8479-a1c3447d484e","Type":"ContainerDied","Data":"5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5"}
Jul 20 10:25:26.719253 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.719232    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:25:26.753275 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.753244    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whhf6\" (UniqueName: \"kubernetes.io/projected/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kube-api-access-whhf6\") pod \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") "
Jul 20 10:25:26.753403 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.753310    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-proxy-tls\") pod \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") "
Jul 20 10:25:26.753403 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.753361    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kserve-provision-location\") pod \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") "
Jul 20 10:25:26.753526 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.753417    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-cabundle-cert\") pod \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") "
Jul 20 10:25:26.753526 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.753447    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kube-rbac-proxy-sar-config\") pod \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\" (UID: \"2ad87b9b-9ae7-43ef-8479-a1c3447d484e\") "
Jul 20 10:25:26.753768 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.753742    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2ad87b9b-9ae7-43ef-8479-a1c3447d484e" (UID: "2ad87b9b-9ae7-43ef-8479-a1c3447d484e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:25:26.753892 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.753854    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ad87b9b-9ae7-43ef-8479-a1c3447d484e" (UID: "2ad87b9b-9ae7-43ef-8479-a1c3447d484e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:25:26.754414 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.754391    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ad87b9b-9ae7-43ef-8479-a1c3447d484e" (UID: "2ad87b9b-9ae7-43ef-8479-a1c3447d484e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:25:26.755357 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.755328    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ad87b9b-9ae7-43ef-8479-a1c3447d484e" (UID: "2ad87b9b-9ae7-43ef-8479-a1c3447d484e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:25:26.755736 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.755719    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kube-api-access-whhf6" (OuterVolumeSpecName: "kube-api-access-whhf6") pod "2ad87b9b-9ae7-43ef-8479-a1c3447d484e" (UID: "2ad87b9b-9ae7-43ef-8479-a1c3447d484e"). InnerVolumeSpecName "kube-api-access-whhf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:25:26.854578 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.854528    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:25:26.854578 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.854545    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:25:26.854578 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.854554    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whhf6\" (UniqueName: \"kubernetes.io/projected/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kube-api-access-whhf6\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:25:26.854578 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.854563    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:25:26.854578 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:26.854572    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ad87b9b-9ae7-43ef-8479-a1c3447d484e-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:25:27.000679 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.000657    2561 generic.go:356] "Generic (PLEG): container finished" podID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerID="3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1" exitCode=0
Jul 20 10:25:27.000784 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.000703    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"
Jul 20 10:25:27.000784 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.000738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" event={"ID":"2ad87b9b-9ae7-43ef-8479-a1c3447d484e","Type":"ContainerDied","Data":"3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1"}
Jul 20 10:25:27.000868 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.000791    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc" event={"ID":"2ad87b9b-9ae7-43ef-8479-a1c3447d484e","Type":"ContainerDied","Data":"7cd4cd5505f75f7f6ca6da3e646c83be92535dbed48c134c088d8bf58ce19f01"}
Jul 20 10:25:27.000868 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.000818    2561 scope.go:117] "RemoveContainer" containerID="5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5"
Jul 20 10:25:27.008453 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.008439    2561 scope.go:117] "RemoveContainer" containerID="3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1"
Jul 20 10:25:27.015171 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.015153    2561 scope.go:117] "RemoveContainer" containerID="42ab5d3925bfb54c206370b51a29126abac4a2c4ec75761b1456b34714f3032d"
Jul 20 10:25:27.021979 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.021966    2561 scope.go:117] "RemoveContainer" containerID="5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5"
Jul 20 10:25:27.022254 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:25:27.022236    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5\": container with ID starting with 5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5 not found: ID does not exist" containerID="5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5"
Jul 20 10:25:27.022322 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.022261    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5"} err="failed to get container status \"5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5\": rpc error: code = NotFound desc = could not find container \"5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5\": container with ID starting with 5ca4b3aefe67eeaa6644cd47437e8e2502b18f56aa61187eb45e6707d6579ca5 not found: ID does not exist"
Jul 20 10:25:27.022322 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.022277    2561 scope.go:117] "RemoveContainer" containerID="3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1"
Jul 20 10:25:27.022495 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:25:27.022478    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1\": container with ID starting with 3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1 not found: ID does not exist" containerID="3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1"
Jul 20 10:25:27.022531 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.022502    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1"} err="failed to get container status \"3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1\": rpc error: code = NotFound desc = could not find container \"3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1\": container with ID starting with 3194912e98852cce3bf0c37e35740c3795e1be894f42daec63ddd856716effe1 not found: ID does not exist"
Jul 20 10:25:27.022531 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.022518    2561 scope.go:117] "RemoveContainer" containerID="42ab5d3925bfb54c206370b51a29126abac4a2c4ec75761b1456b34714f3032d"
Jul 20 10:25:27.022743 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:25:27.022728    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ab5d3925bfb54c206370b51a29126abac4a2c4ec75761b1456b34714f3032d\": container with ID starting with 42ab5d3925bfb54c206370b51a29126abac4a2c4ec75761b1456b34714f3032d not found: ID does not exist" containerID="42ab5d3925bfb54c206370b51a29126abac4a2c4ec75761b1456b34714f3032d"
Jul 20 10:25:27.022786 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.022747    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ab5d3925bfb54c206370b51a29126abac4a2c4ec75761b1456b34714f3032d"} err="failed to get container status \"42ab5d3925bfb54c206370b51a29126abac4a2c4ec75761b1456b34714f3032d\": rpc error: code = NotFound desc = could not find container \"42ab5d3925bfb54c206370b51a29126abac4a2c4ec75761b1456b34714f3032d\": container with ID starting with 42ab5d3925bfb54c206370b51a29126abac4a2c4ec75761b1456b34714f3032d not found: ID does not exist"
Jul 20 10:25:27.041615 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.041595    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"]
Jul 20 10:25:27.048839 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:27.048820    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-lp9jc"]
Jul 20 10:25:28.766150 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:28.766094    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" path="/var/lib/kubelet/pods/2ad87b9b-9ae7-43ef-8479-a1c3447d484e/volumes"
Jul 20 10:25:30.360504 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.360471    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"]
Jul 20 10:25:30.360863 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.360719    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="kserve-container"
Jul 20 10:25:30.360863 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.360729    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="kserve-container"
Jul 20 10:25:30.360863 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.360740    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="kube-rbac-proxy"
Jul 20 10:25:30.360863 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.360746    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="kube-rbac-proxy"
Jul 20 10:25:30.360863 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.360758    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="storage-initializer"
Jul 20 10:25:30.360863 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.360763    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="storage-initializer"
Jul 20 10:25:30.360863 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.360809    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="kube-rbac-proxy"
Jul 20 10:25:30.360863 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.360816    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ad87b9b-9ae7-43ef-8479-a1c3447d484e" containerName="kserve-container"
Jul 20 10:25:30.365689 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.365665    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.368538 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.368510    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:25:30.368642 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.368581    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:25:30.368701 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.368658    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 20 10:25:30.368749 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.368719    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 20 10:25:30.368792 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.368777    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:25:30.369289 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.369272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:25:30.369368 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.369325    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:25:30.378160 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.378139    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"]
Jul 20 10:25:30.483039 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.483004    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.483236 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.483043    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.483236 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.483069    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z2lj\" (UniqueName: \"kubernetes.io/projected/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kube-api-access-5z2lj\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.483236 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.483190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.483380 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.483236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.583855 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.583824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.584027 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.583865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.584027 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.583994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z2lj\" (UniqueName: \"kubernetes.io/projected/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kube-api-access-5z2lj\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.584167 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.584056    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.584167 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.584093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.584606 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.584207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.584761 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.584740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.584847 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.584809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.586590 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.586573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.594450 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.594425    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z2lj\" (UniqueName: \"kubernetes.io/projected/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kube-api-access-5z2lj\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.675787 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.675715    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:30.804559 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:30.804535    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"]
Jul 20 10:25:30.806814 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:25:30.806792    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebdeb4b_fcc9_4117_abb8_097918ce2ee2.slice/crio-cc34a076bd744ef29571a828b038b96b9119412c92fb5b4a2fc875bd7d443752 WatchSource:0}: Error finding container cc34a076bd744ef29571a828b038b96b9119412c92fb5b4a2fc875bd7d443752: Status 404 returned error can't find the container with id cc34a076bd744ef29571a828b038b96b9119412c92fb5b4a2fc875bd7d443752
Jul 20 10:25:31.020920 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:31.020838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" event={"ID":"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2","Type":"ContainerStarted","Data":"1c93d56dcfaf51626281286f0962c67d681b569d19e98ab6d300c471be6fb5cb"}
Jul 20 10:25:31.020920 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:31.020899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" event={"ID":"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2","Type":"ContainerStarted","Data":"cc34a076bd744ef29571a828b038b96b9119412c92fb5b4a2fc875bd7d443752"}
Jul 20 10:25:40.056731 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:40.056695    2561 generic.go:356] "Generic (PLEG): container finished" podID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerID="1c93d56dcfaf51626281286f0962c67d681b569d19e98ab6d300c471be6fb5cb" exitCode=0
Jul 20 10:25:40.057212 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:40.056776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" event={"ID":"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2","Type":"ContainerDied","Data":"1c93d56dcfaf51626281286f0962c67d681b569d19e98ab6d300c471be6fb5cb"}
Jul 20 10:25:41.063186 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:41.063153    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" event={"ID":"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2","Type":"ContainerStarted","Data":"dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061"}
Jul 20 10:25:41.063583 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:41.063222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" event={"ID":"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2","Type":"ContainerStarted","Data":"92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca"}
Jul 20 10:25:41.063583 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:41.063416    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:41.084356 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:41.084310    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" podStartSLOduration=2.905413749 podStartE2EDuration="11.084296078s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.178882329s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:25:41.083407083 +0000 UTC m=+1720.921505704" watchObservedRunningTime="2026-07-20 10:25:41.084296078 +0000 UTC m=+1720.922394662"
Jul 20 10:25:42.068176 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:42.068140    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:42.069473 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:42.069445    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 20 10:25:43.071443 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:43.071399    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 20 10:25:48.076825 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:48.076792    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:25:48.077486 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:48.077457    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 20 10:25:58.078835 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:25:58.078801    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:26:05.471270 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:05.471236    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"]
Jul 20 10:26:05.471712 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:05.471564    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="kserve-container" containerID="cri-o://92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca" gracePeriod=30
Jul 20 10:26:05.471712 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:05.471573    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="kube-rbac-proxy" containerID="cri-o://dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061" gracePeriod=30
Jul 20 10:26:06.161809 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.161779    2561 generic.go:356] "Generic (PLEG): container finished" podID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerID="dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061" exitCode=2
Jul 20 10:26:06.161986 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.161822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" event={"ID":"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2","Type":"ContainerDied","Data":"dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061"}
Jul 20 10:26:06.706287 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.706264    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:26:06.857137 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.857053    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kube-rbac-proxy-sar-config\") pod \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") "
Jul 20 10:26:06.857137 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.857084    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z2lj\" (UniqueName: \"kubernetes.io/projected/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kube-api-access-5z2lj\") pod \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") "
Jul 20 10:26:06.857137 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.857106    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-proxy-tls\") pod \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") "
Jul 20 10:26:06.857407 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.857258    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kserve-provision-location\") pod \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") "
Jul 20 10:26:06.857407 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.857309    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-cabundle-cert\") pod \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\" (UID: \"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2\") "
Jul 20 10:26:06.857585 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.857553    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" (UID: "8ebdeb4b-fcc9-4117-abb8-097918ce2ee2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:26:06.857718 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.857658    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" (UID: "8ebdeb4b-fcc9-4117-abb8-097918ce2ee2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:26:06.858447 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.858419    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" (UID: "8ebdeb4b-fcc9-4117-abb8-097918ce2ee2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:26:06.859259 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.859238    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kube-api-access-5z2lj" (OuterVolumeSpecName: "kube-api-access-5z2lj") pod "8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" (UID: "8ebdeb4b-fcc9-4117-abb8-097918ce2ee2"). InnerVolumeSpecName "kube-api-access-5z2lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:26:06.859346 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.859304    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" (UID: "8ebdeb4b-fcc9-4117-abb8-097918ce2ee2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:26:06.958727 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.958705    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:26:06.958727 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.958725    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:26:06.958852 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.958735    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:26:06.958852 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.958744    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z2lj\" (UniqueName: \"kubernetes.io/projected/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-kube-api-access-5z2lj\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:26:06.958852 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:06.958753    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:26:07.166848 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.166817    2561 generic.go:356] "Generic (PLEG): container finished" podID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerID="92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca" exitCode=0
Jul 20 10:26:07.167001 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.166881    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"
Jul 20 10:26:07.167001 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.166904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" event={"ID":"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2","Type":"ContainerDied","Data":"92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca"}
Jul 20 10:26:07.167001 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.166971    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z" event={"ID":"8ebdeb4b-fcc9-4117-abb8-097918ce2ee2","Type":"ContainerDied","Data":"cc34a076bd744ef29571a828b038b96b9119412c92fb5b4a2fc875bd7d443752"}
Jul 20 10:26:07.167001 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.166998    2561 scope.go:117] "RemoveContainer" containerID="dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061"
Jul 20 10:26:07.175104 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.175083    2561 scope.go:117] "RemoveContainer" containerID="92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca"
Jul 20 10:26:07.182013 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.181998    2561 scope.go:117] "RemoveContainer" containerID="1c93d56dcfaf51626281286f0962c67d681b569d19e98ab6d300c471be6fb5cb"
Jul 20 10:26:07.191363 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.191307    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"]
Jul 20 10:26:07.191713 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.191679    2561 scope.go:117] "RemoveContainer" containerID="dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061"
Jul 20 10:26:07.192568 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:26:07.192040    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061\": container with ID starting with dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061 not found: ID does not exist" containerID="dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061"
Jul 20 10:26:07.192568 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.192074    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061"} err="failed to get container status \"dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061\": rpc error: code = NotFound desc = could not find container \"dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061\": container with ID starting with dc2f6cb96effc1331de58f4bdcaca3222c24422d1f67ae2c105f6f61a4543061 not found: ID does not exist"
Jul 20 10:26:07.192568 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.192097    2561 scope.go:117] "RemoveContainer" containerID="92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca"
Jul 20 10:26:07.192568 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:26:07.192449    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca\": container with ID starting with 92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca not found: ID does not exist" containerID="92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca"
Jul 20 10:26:07.192568 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.192473    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca"} err="failed to get container status \"92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca\": rpc error: code = NotFound desc = could not find container \"92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca\": container with ID starting with 92af9bc0a66bef105c490f9b00d152196bbb2129c6293f15e2bb8e38192069ca not found: ID does not exist"
Jul 20 10:26:07.192568 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.192509    2561 scope.go:117] "RemoveContainer" containerID="1c93d56dcfaf51626281286f0962c67d681b569d19e98ab6d300c471be6fb5cb"
Jul 20 10:26:07.192899 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:26:07.192795    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c93d56dcfaf51626281286f0962c67d681b569d19e98ab6d300c471be6fb5cb\": container with ID starting with 1c93d56dcfaf51626281286f0962c67d681b569d19e98ab6d300c471be6fb5cb not found: ID does not exist" containerID="1c93d56dcfaf51626281286f0962c67d681b569d19e98ab6d300c471be6fb5cb"
Jul 20 10:26:07.192899 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.192820    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c93d56dcfaf51626281286f0962c67d681b569d19e98ab6d300c471be6fb5cb"} err="failed to get container status \"1c93d56dcfaf51626281286f0962c67d681b569d19e98ab6d300c471be6fb5cb\": rpc error: code = NotFound desc = could not find container \"1c93d56dcfaf51626281286f0962c67d681b569d19e98ab6d300c471be6fb5cb\": container with ID starting with 1c93d56dcfaf51626281286f0962c67d681b569d19e98ab6d300c471be6fb5cb not found: ID does not exist"
Jul 20 10:26:07.193579 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:07.193559    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qt5z"]
Jul 20 10:26:08.765021 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:08.764991    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" path="/var/lib/kubelet/pods/8ebdeb4b-fcc9-4117-abb8-097918ce2ee2/volumes"
Jul 20 10:26:10.666289 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.666253    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"]
Jul 20 10:26:10.666677 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.666524    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="kserve-container"
Jul 20 10:26:10.666677 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.666534    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="kserve-container"
Jul 20 10:26:10.666677 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.666547    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="kube-rbac-proxy"
Jul 20 10:26:10.666677 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.666552    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="kube-rbac-proxy"
Jul 20 10:26:10.666677 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.666560    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="storage-initializer"
Jul 20 10:26:10.666677 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.666566    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="storage-initializer"
Jul 20 10:26:10.666677 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.666613    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="kube-rbac-proxy"
Jul 20 10:26:10.666677 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.666624    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ebdeb4b-fcc9-4117-abb8-097918ce2ee2" containerName="kserve-container"
Jul 20 10:26:10.671041 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.671020    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.673548 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.673505    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 20 10:26:10.673711 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.673544    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:26:10.673711 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.673559    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 20 10:26:10.673865 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.673849    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:26:10.674512 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.674486    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:26:10.674614 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.674559    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:26:10.674659 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.674618    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:26:10.680914 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.680872    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"]
Jul 20 10:26:10.783447 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.783419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dhhn\" (UniqueName: \"kubernetes.io/projected/aa23dfca-04db-42f6-aff4-f5af13b37648-kube-api-access-2dhhn\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.783447 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.783446    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa23dfca-04db-42f6-aff4-f5af13b37648-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.783631 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.783483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa23dfca-04db-42f6-aff4-f5af13b37648-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.783631 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.783555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa23dfca-04db-42f6-aff4-f5af13b37648-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.783631 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.783583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa23dfca-04db-42f6-aff4-f5af13b37648-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.884310 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.884278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhhn\" (UniqueName: \"kubernetes.io/projected/aa23dfca-04db-42f6-aff4-f5af13b37648-kube-api-access-2dhhn\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.884310 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.884309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa23dfca-04db-42f6-aff4-f5af13b37648-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.884480 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.884387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa23dfca-04db-42f6-aff4-f5af13b37648-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.884480 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.884426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa23dfca-04db-42f6-aff4-f5af13b37648-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.884480 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.884450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa23dfca-04db-42f6-aff4-f5af13b37648-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.884624 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:26:10.884566    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Jul 20 10:26:10.884678 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:26:10.884644    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa23dfca-04db-42f6-aff4-f5af13b37648-proxy-tls podName:aa23dfca-04db-42f6-aff4-f5af13b37648 nodeName:}" failed. No retries permitted until 2026-07-20 10:26:11.384624132 +0000 UTC m=+1751.222722700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aa23dfca-04db-42f6-aff4-f5af13b37648-proxy-tls") pod "isvc-lightgbm-predictor-54f85ddf88-hptvv" (UID: "aa23dfca-04db-42f6-aff4-f5af13b37648") : secret "isvc-lightgbm-predictor-serving-cert" not found
Jul 20 10:26:10.884777 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.884759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa23dfca-04db-42f6-aff4-f5af13b37648-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.884983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.884966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa23dfca-04db-42f6-aff4-f5af13b37648-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.885035 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.885019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa23dfca-04db-42f6-aff4-f5af13b37648-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:10.893942 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:10.893920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhhn\" (UniqueName: \"kubernetes.io/projected/aa23dfca-04db-42f6-aff4-f5af13b37648-kube-api-access-2dhhn\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:11.388845 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:11.388810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa23dfca-04db-42f6-aff4-f5af13b37648-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:11.391107 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:11.391087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa23dfca-04db-42f6-aff4-f5af13b37648-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-hptvv\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:11.582987 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:11.582958    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:11.713603 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:11.713578    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"]
Jul 20 10:26:11.716349 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:26:11.716323    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa23dfca_04db_42f6_aff4_f5af13b37648.slice/crio-446b6e480f44ca47c4336bae103c8b170c6beddb8508e9d9bbb3ba1bd67032ef WatchSource:0}: Error finding container 446b6e480f44ca47c4336bae103c8b170c6beddb8508e9d9bbb3ba1bd67032ef: Status 404 returned error can't find the container with id 446b6e480f44ca47c4336bae103c8b170c6beddb8508e9d9bbb3ba1bd67032ef
Jul 20 10:26:12.192904 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:12.192869    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" event={"ID":"aa23dfca-04db-42f6-aff4-f5af13b37648","Type":"ContainerStarted","Data":"5b4022517eb306ccfa82d72a6ca311381c54e972361d5511f1a8ab9513b37932"}
Jul 20 10:26:12.193105 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:12.192939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" event={"ID":"aa23dfca-04db-42f6-aff4-f5af13b37648","Type":"ContainerStarted","Data":"446b6e480f44ca47c4336bae103c8b170c6beddb8508e9d9bbb3ba1bd67032ef"}
Jul 20 10:26:16.210172 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:16.210140    2561 generic.go:356] "Generic (PLEG): container finished" podID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerID="5b4022517eb306ccfa82d72a6ca311381c54e972361d5511f1a8ab9513b37932" exitCode=0
Jul 20 10:26:16.210630 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:16.210212    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" event={"ID":"aa23dfca-04db-42f6-aff4-f5af13b37648","Type":"ContainerDied","Data":"5b4022517eb306ccfa82d72a6ca311381c54e972361d5511f1a8ab9513b37932"}
Jul 20 10:26:16.211497 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:16.211482    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:26:23.240182 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:23.240148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" event={"ID":"aa23dfca-04db-42f6-aff4-f5af13b37648","Type":"ContainerStarted","Data":"de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072"}
Jul 20 10:26:23.240554 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:23.240206    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" event={"ID":"aa23dfca-04db-42f6-aff4-f5af13b37648","Type":"ContainerStarted","Data":"035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528"}
Jul 20 10:26:23.240554 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:23.240383    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:23.263642 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:23.263597    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" podStartSLOduration=3.456941565 podStartE2EDuration="13.263580929s" totalImagesPullingTime="6.196226227s" totalInitContainerRuntime="3.610413137s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:26:23.261738656 +0000 UTC m=+1763.099837271" watchObservedRunningTime="2026-07-20 10:26:23.263580929 +0000 UTC m=+1763.101679514"
Jul 20 10:26:24.244340 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:24.244313    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:24.245460 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:24.245434    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 20 10:26:25.247610 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:25.247574    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 20 10:26:30.252661 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:30.252631    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:26:30.253181 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:30.253143    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 20 10:26:40.253312 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:40.253270    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 20 10:26:50.253774 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:26:50.253735    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 20 10:27:00.253436 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:00.253392    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 20 10:27:10.253454 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:10.253416    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 20 10:27:16.053878 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:16.053852    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:27:16.056224 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:16.056201    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:27:20.254025 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:20.253986    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 20 10:27:30.253522 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:30.253478    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 20 10:27:40.254271 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:40.254243    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:27:40.823857 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:40.823822    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"]
Jul 20 10:27:40.824054 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:40.824033    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container" containerID="cri-o://035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528" gracePeriod=30
Jul 20 10:27:40.824146 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:40.824071    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kube-rbac-proxy" containerID="cri-o://de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072" gracePeriod=30
Jul 20 10:27:41.518895 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:41.518865    2561 generic.go:356] "Generic (PLEG): container finished" podID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerID="de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072" exitCode=2
Jul 20 10:27:41.519290 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:41.518934    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" event={"ID":"aa23dfca-04db-42f6-aff4-f5af13b37648","Type":"ContainerDied","Data":"de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072"}
Jul 20 10:27:44.767278 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.767256    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:27:44.875949 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.875928    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dhhn\" (UniqueName: \"kubernetes.io/projected/aa23dfca-04db-42f6-aff4-f5af13b37648-kube-api-access-2dhhn\") pod \"aa23dfca-04db-42f6-aff4-f5af13b37648\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") "
Jul 20 10:27:44.876095 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.875965    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa23dfca-04db-42f6-aff4-f5af13b37648-proxy-tls\") pod \"aa23dfca-04db-42f6-aff4-f5af13b37648\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") "
Jul 20 10:27:44.876095 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.875995    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa23dfca-04db-42f6-aff4-f5af13b37648-kube-rbac-proxy-sar-config\") pod \"aa23dfca-04db-42f6-aff4-f5af13b37648\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") "
Jul 20 10:27:44.876095 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.876028    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa23dfca-04db-42f6-aff4-f5af13b37648-cabundle-cert\") pod \"aa23dfca-04db-42f6-aff4-f5af13b37648\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") "
Jul 20 10:27:44.876095 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.876063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa23dfca-04db-42f6-aff4-f5af13b37648-kserve-provision-location\") pod \"aa23dfca-04db-42f6-aff4-f5af13b37648\" (UID: \"aa23dfca-04db-42f6-aff4-f5af13b37648\") "
Jul 20 10:27:44.876462 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.876422    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa23dfca-04db-42f6-aff4-f5af13b37648-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aa23dfca-04db-42f6-aff4-f5af13b37648" (UID: "aa23dfca-04db-42f6-aff4-f5af13b37648"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:27:44.876462 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.876431    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa23dfca-04db-42f6-aff4-f5af13b37648-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa23dfca-04db-42f6-aff4-f5af13b37648" (UID: "aa23dfca-04db-42f6-aff4-f5af13b37648"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:27:44.876462 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.876446    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa23dfca-04db-42f6-aff4-f5af13b37648-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aa23dfca-04db-42f6-aff4-f5af13b37648" (UID: "aa23dfca-04db-42f6-aff4-f5af13b37648"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:27:44.877975 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.877955    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa23dfca-04db-42f6-aff4-f5af13b37648-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa23dfca-04db-42f6-aff4-f5af13b37648" (UID: "aa23dfca-04db-42f6-aff4-f5af13b37648"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:27:44.878056 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.877999    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa23dfca-04db-42f6-aff4-f5af13b37648-kube-api-access-2dhhn" (OuterVolumeSpecName: "kube-api-access-2dhhn") pod "aa23dfca-04db-42f6-aff4-f5af13b37648" (UID: "aa23dfca-04db-42f6-aff4-f5af13b37648"). InnerVolumeSpecName "kube-api-access-2dhhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:27:44.976951 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.976931    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa23dfca-04db-42f6-aff4-f5af13b37648-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:27:44.976951 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.976950    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa23dfca-04db-42f6-aff4-f5af13b37648-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:27:44.977080 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.976961    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dhhn\" (UniqueName: \"kubernetes.io/projected/aa23dfca-04db-42f6-aff4-f5af13b37648-kube-api-access-2dhhn\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:27:44.977080 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.976970    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa23dfca-04db-42f6-aff4-f5af13b37648-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:27:44.977080 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:44.976979    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa23dfca-04db-42f6-aff4-f5af13b37648-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:27:45.535015 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.534984    2561 generic.go:356] "Generic (PLEG): container finished" podID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerID="035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528" exitCode=0
Jul 20 10:27:45.535209 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.535039    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"
Jul 20 10:27:45.535209 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.535056    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" event={"ID":"aa23dfca-04db-42f6-aff4-f5af13b37648","Type":"ContainerDied","Data":"035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528"}
Jul 20 10:27:45.535209 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.535103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv" event={"ID":"aa23dfca-04db-42f6-aff4-f5af13b37648","Type":"ContainerDied","Data":"446b6e480f44ca47c4336bae103c8b170c6beddb8508e9d9bbb3ba1bd67032ef"}
Jul 20 10:27:45.535209 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.535156    2561 scope.go:117] "RemoveContainer" containerID="de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072"
Jul 20 10:27:45.543729 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.543706    2561 scope.go:117] "RemoveContainer" containerID="035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528"
Jul 20 10:27:45.550647 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.550632    2561 scope.go:117] "RemoveContainer" containerID="5b4022517eb306ccfa82d72a6ca311381c54e972361d5511f1a8ab9513b37932"
Jul 20 10:27:45.556977 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.556957    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"]
Jul 20 10:27:45.558099 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.558088    2561 scope.go:117] "RemoveContainer" containerID="de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072"
Jul 20 10:27:45.558372 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:27:45.558356    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072\": container with ID starting with de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072 not found: ID does not exist" containerID="de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072"
Jul 20 10:27:45.558432 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.558379    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072"} err="failed to get container status \"de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072\": rpc error: code = NotFound desc = could not find container \"de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072\": container with ID starting with de864735d762ab68fc9bb858986daa8e1d6ea3749499c0a77c45a7aadd482072 not found: ID does not exist"
Jul 20 10:27:45.558432 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.558396    2561 scope.go:117] "RemoveContainer" containerID="035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528"
Jul 20 10:27:45.558650 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:27:45.558631    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528\": container with ID starting with 035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528 not found: ID does not exist" containerID="035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528"
Jul 20 10:27:45.558719 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.558658    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528"} err="failed to get container status \"035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528\": rpc error: code = NotFound desc = could not find container \"035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528\": container with ID starting with 035f42c4c275ddcc95d98f2a9871073649cf8dfd1734ba717cb8eab857ee9528 not found: ID does not exist"
Jul 20 10:27:45.558719 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.558681    2561 scope.go:117] "RemoveContainer" containerID="5b4022517eb306ccfa82d72a6ca311381c54e972361d5511f1a8ab9513b37932"
Jul 20 10:27:45.558904 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:27:45.558885    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b4022517eb306ccfa82d72a6ca311381c54e972361d5511f1a8ab9513b37932\": container with ID starting with 5b4022517eb306ccfa82d72a6ca311381c54e972361d5511f1a8ab9513b37932 not found: ID does not exist" containerID="5b4022517eb306ccfa82d72a6ca311381c54e972361d5511f1a8ab9513b37932"
Jul 20 10:27:45.558943 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.558911    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b4022517eb306ccfa82d72a6ca311381c54e972361d5511f1a8ab9513b37932"} err="failed to get container status \"5b4022517eb306ccfa82d72a6ca311381c54e972361d5511f1a8ab9513b37932\": rpc error: code = NotFound desc = could not find container \"5b4022517eb306ccfa82d72a6ca311381c54e972361d5511f1a8ab9513b37932\": container with ID starting with 5b4022517eb306ccfa82d72a6ca311381c54e972361d5511f1a8ab9513b37932 not found: ID does not exist"
Jul 20 10:27:45.563010 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:45.562990    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-hptvv"]
Jul 20 10:27:46.034410 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.034377    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"]
Jul 20 10:27:46.034770 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.034618    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kube-rbac-proxy"
Jul 20 10:27:46.034770 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.034628    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kube-rbac-proxy"
Jul 20 10:27:46.034770 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.034642    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="storage-initializer"
Jul 20 10:27:46.034770 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.034649    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="storage-initializer"
Jul 20 10:27:46.034770 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.034655    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container"
Jul 20 10:27:46.034770 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.034661    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container"
Jul 20 10:27:46.034770 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.034706    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kube-rbac-proxy"
Jul 20 10:27:46.034770 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.034715    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" containerName="kserve-container"
Jul 20 10:27:46.038806 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.038785    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.041938 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.041915    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:27:46.041938 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.041932    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 20 10:27:46.042209 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.042194    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 20 10:27:46.042508 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.042487    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:27:46.042631 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.042568    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:27:46.043091 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.043077    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:27:46.045346 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.045328    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:27:46.056812 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.056792    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"]
Jul 20 10:27:46.186664 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.186618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/590e3a12-757b-4f88-aaaf-38d16d927996-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.186869 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.186684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdckx\" (UniqueName: \"kubernetes.io/projected/590e3a12-757b-4f88-aaaf-38d16d927996-kube-api-access-cdckx\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.186869 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.186780    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/590e3a12-757b-4f88-aaaf-38d16d927996-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.186869 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.186818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/590e3a12-757b-4f88-aaaf-38d16d927996-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.187017 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.186889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/590e3a12-757b-4f88-aaaf-38d16d927996-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.287922 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.287826    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/590e3a12-757b-4f88-aaaf-38d16d927996-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.287922 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.287885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdckx\" (UniqueName: \"kubernetes.io/projected/590e3a12-757b-4f88-aaaf-38d16d927996-kube-api-access-cdckx\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.287922 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.287908    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/590e3a12-757b-4f88-aaaf-38d16d927996-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.287922 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.287924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/590e3a12-757b-4f88-aaaf-38d16d927996-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.288285 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.287954    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/590e3a12-757b-4f88-aaaf-38d16d927996-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.288421 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.288401    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/590e3a12-757b-4f88-aaaf-38d16d927996-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.288548 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.288531    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/590e3a12-757b-4f88-aaaf-38d16d927996-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.288588 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.288573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/590e3a12-757b-4f88-aaaf-38d16d927996-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.290399 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.290374    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/590e3a12-757b-4f88-aaaf-38d16d927996-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.296569 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.296546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdckx\" (UniqueName: \"kubernetes.io/projected/590e3a12-757b-4f88-aaaf-38d16d927996-kube-api-access-cdckx\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.348458 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.348430    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:46.491730 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.491580    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"]
Jul 20 10:27:46.540767 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.540713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" event={"ID":"590e3a12-757b-4f88-aaaf-38d16d927996","Type":"ContainerStarted","Data":"fc4840fef9cbe364f8592beeb623ca249b4fc740ef80f850e0c32b03e8b57b63"}
Jul 20 10:27:46.765439 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:46.765406    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa23dfca-04db-42f6-aff4-f5af13b37648" path="/var/lib/kubelet/pods/aa23dfca-04db-42f6-aff4-f5af13b37648/volumes"
Jul 20 10:27:47.545496 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:47.545459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" event={"ID":"590e3a12-757b-4f88-aaaf-38d16d927996","Type":"ContainerStarted","Data":"071b3b27bbf87607cbb5f667f8d6cb854133ec0238e1c56e149bc3bbd404d22d"}
Jul 20 10:27:50.558065 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:50.557899    2561 generic.go:356] "Generic (PLEG): container finished" podID="590e3a12-757b-4f88-aaaf-38d16d927996" containerID="071b3b27bbf87607cbb5f667f8d6cb854133ec0238e1c56e149bc3bbd404d22d" exitCode=0
Jul 20 10:27:50.558065 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:50.557975    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" event={"ID":"590e3a12-757b-4f88-aaaf-38d16d927996","Type":"ContainerDied","Data":"071b3b27bbf87607cbb5f667f8d6cb854133ec0238e1c56e149bc3bbd404d22d"}
Jul 20 10:27:51.563976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:51.563945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" event={"ID":"590e3a12-757b-4f88-aaaf-38d16d927996","Type":"ContainerStarted","Data":"6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541"}
Jul 20 10:27:51.564497 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:51.563998    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" event={"ID":"590e3a12-757b-4f88-aaaf-38d16d927996","Type":"ContainerStarted","Data":"8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773"}
Jul 20 10:27:51.564497 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:51.564287    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:51.564497 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:51.564416    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:51.565730 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:51.565706    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 20 10:27:51.589534 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:51.589481    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podStartSLOduration=2.960193443 podStartE2EDuration="6.589468212s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.629274769s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:27:51.586606408 +0000 UTC m=+1851.424705020" watchObservedRunningTime="2026-07-20 10:27:51.589468212 +0000 UTC m=+1851.427566796"
Jul 20 10:27:52.568238 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:52.568194    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 20 10:27:57.572994 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:57.572962    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:27:57.573464 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:27:57.573439    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 20 10:28:07.573418 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:28:07.573381    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 20 10:28:17.573404 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:28:17.573363    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 20 10:28:27.573682 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:28:27.573595    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 20 10:28:37.573594 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:28:37.573556    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 20 10:28:47.573701 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:28:47.573665    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 20 10:28:57.574069 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:28:57.574027    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 20 10:29:07.574823 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:07.574793    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:29:16.546834 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:16.546805    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"]
Jul 20 10:29:16.547247 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:16.547030    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container" containerID="cri-o://8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773" gracePeriod=30
Jul 20 10:29:16.547247 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:16.547143    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kube-rbac-proxy" containerID="cri-o://6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541" gracePeriod=30
Jul 20 10:29:16.867740 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:16.867709    2561 generic.go:356] "Generic (PLEG): container finished" podID="590e3a12-757b-4f88-aaaf-38d16d927996" containerID="6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541" exitCode=2
Jul 20 10:29:16.867909 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:16.867784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" event={"ID":"590e3a12-757b-4f88-aaaf-38d16d927996","Type":"ContainerDied","Data":"6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541"}
Jul 20 10:29:17.569258 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:17.569217    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 20 10:29:17.573702 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:17.573677    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 20 10:29:20.795900 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.795876    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:29:20.883480 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.883447    2561 generic.go:356] "Generic (PLEG): container finished" podID="590e3a12-757b-4f88-aaaf-38d16d927996" containerID="8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773" exitCode=0
Jul 20 10:29:20.883638 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.883511    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"
Jul 20 10:29:20.883638 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.883535    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" event={"ID":"590e3a12-757b-4f88-aaaf-38d16d927996","Type":"ContainerDied","Data":"8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773"}
Jul 20 10:29:20.883638 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.883606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w" event={"ID":"590e3a12-757b-4f88-aaaf-38d16d927996","Type":"ContainerDied","Data":"fc4840fef9cbe364f8592beeb623ca249b4fc740ef80f850e0c32b03e8b57b63"}
Jul 20 10:29:20.883806 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.883644    2561 scope.go:117] "RemoveContainer" containerID="6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541"
Jul 20 10:29:20.891001 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.890986    2561 scope.go:117] "RemoveContainer" containerID="8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773"
Jul 20 10:29:20.897855 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.897838    2561 scope.go:117] "RemoveContainer" containerID="071b3b27bbf87607cbb5f667f8d6cb854133ec0238e1c56e149bc3bbd404d22d"
Jul 20 10:29:20.904317 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.904304    2561 scope.go:117] "RemoveContainer" containerID="6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541"
Jul 20 10:29:20.904560 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:29:20.904543    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541\": container with ID starting with 6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541 not found: ID does not exist" containerID="6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541"
Jul 20 10:29:20.904623 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.904567    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541"} err="failed to get container status \"6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541\": rpc error: code = NotFound desc = could not find container \"6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541\": container with ID starting with 6154750d881ee71282ef44ad667810f6bd0729b372216eaf588b67fdf3ef6541 not found: ID does not exist"
Jul 20 10:29:20.904623 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.904583    2561 scope.go:117] "RemoveContainer" containerID="8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773"
Jul 20 10:29:20.904863 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:29:20.904846    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773\": container with ID starting with 8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773 not found: ID does not exist" containerID="8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773"
Jul 20 10:29:20.904918 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.904868    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773"} err="failed to get container status \"8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773\": rpc error: code = NotFound desc = could not find container \"8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773\": container with ID starting with 8e352f8b0101236325f85904dd259c5833d8a8f2299017e28e8ca296e84eb773 not found: ID does not exist"
Jul 20 10:29:20.904918 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.904883    2561 scope.go:117] "RemoveContainer" containerID="071b3b27bbf87607cbb5f667f8d6cb854133ec0238e1c56e149bc3bbd404d22d"
Jul 20 10:29:20.905154 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:29:20.905140    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071b3b27bbf87607cbb5f667f8d6cb854133ec0238e1c56e149bc3bbd404d22d\": container with ID starting with 071b3b27bbf87607cbb5f667f8d6cb854133ec0238e1c56e149bc3bbd404d22d not found: ID does not exist" containerID="071b3b27bbf87607cbb5f667f8d6cb854133ec0238e1c56e149bc3bbd404d22d"
Jul 20 10:29:20.905204 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.905156    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071b3b27bbf87607cbb5f667f8d6cb854133ec0238e1c56e149bc3bbd404d22d"} err="failed to get container status \"071b3b27bbf87607cbb5f667f8d6cb854133ec0238e1c56e149bc3bbd404d22d\": rpc error: code = NotFound desc = could not find container \"071b3b27bbf87607cbb5f667f8d6cb854133ec0238e1c56e149bc3bbd404d22d\": container with ID starting with 071b3b27bbf87607cbb5f667f8d6cb854133ec0238e1c56e149bc3bbd404d22d not found: ID does not exist"
Jul 20 10:29:20.970957 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.970936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdckx\" (UniqueName: \"kubernetes.io/projected/590e3a12-757b-4f88-aaaf-38d16d927996-kube-api-access-cdckx\") pod \"590e3a12-757b-4f88-aaaf-38d16d927996\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") "
Jul 20 10:29:20.971046 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.970971    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/590e3a12-757b-4f88-aaaf-38d16d927996-kube-rbac-proxy-sar-config\") pod \"590e3a12-757b-4f88-aaaf-38d16d927996\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") "
Jul 20 10:29:20.971046 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.971011    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/590e3a12-757b-4f88-aaaf-38d16d927996-kserve-provision-location\") pod \"590e3a12-757b-4f88-aaaf-38d16d927996\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") "
Jul 20 10:29:20.971186 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.971142    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/590e3a12-757b-4f88-aaaf-38d16d927996-cabundle-cert\") pod \"590e3a12-757b-4f88-aaaf-38d16d927996\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") "
Jul 20 10:29:20.971248 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.971197    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/590e3a12-757b-4f88-aaaf-38d16d927996-proxy-tls\") pod \"590e3a12-757b-4f88-aaaf-38d16d927996\" (UID: \"590e3a12-757b-4f88-aaaf-38d16d927996\") "
Jul 20 10:29:20.971413 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.971390    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590e3a12-757b-4f88-aaaf-38d16d927996-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "590e3a12-757b-4f88-aaaf-38d16d927996" (UID: "590e3a12-757b-4f88-aaaf-38d16d927996"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:29:20.971519 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.971420    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590e3a12-757b-4f88-aaaf-38d16d927996-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "590e3a12-757b-4f88-aaaf-38d16d927996" (UID: "590e3a12-757b-4f88-aaaf-38d16d927996"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:29:20.971568 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.971521    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590e3a12-757b-4f88-aaaf-38d16d927996-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "590e3a12-757b-4f88-aaaf-38d16d927996" (UID: "590e3a12-757b-4f88-aaaf-38d16d927996"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:29:20.972875 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.972848    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590e3a12-757b-4f88-aaaf-38d16d927996-kube-api-access-cdckx" (OuterVolumeSpecName: "kube-api-access-cdckx") pod "590e3a12-757b-4f88-aaaf-38d16d927996" (UID: "590e3a12-757b-4f88-aaaf-38d16d927996"). InnerVolumeSpecName "kube-api-access-cdckx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:29:20.973058 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:20.973041    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590e3a12-757b-4f88-aaaf-38d16d927996-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "590e3a12-757b-4f88-aaaf-38d16d927996" (UID: "590e3a12-757b-4f88-aaaf-38d16d927996"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:29:21.071898 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.071870    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/590e3a12-757b-4f88-aaaf-38d16d927996-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:29:21.071898 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.071894    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/590e3a12-757b-4f88-aaaf-38d16d927996-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:29:21.072043 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.071906    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdckx\" (UniqueName: \"kubernetes.io/projected/590e3a12-757b-4f88-aaaf-38d16d927996-kube-api-access-cdckx\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:29:21.072043 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.071921    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/590e3a12-757b-4f88-aaaf-38d16d927996-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:29:21.072043 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.071933    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/590e3a12-757b-4f88-aaaf-38d16d927996-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:29:21.204800 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.204770    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"]
Jul 20 10:29:21.208938 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.208918    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rrf7w"]
Jul 20 10:29:21.733278 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.733250    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"]
Jul 20 10:29:21.733512 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.733499    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kube-rbac-proxy"
Jul 20 10:29:21.733512 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.733512    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kube-rbac-proxy"
Jul 20 10:29:21.733625 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.733531    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="storage-initializer"
Jul 20 10:29:21.733625 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.733537    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="storage-initializer"
Jul 20 10:29:21.733625 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.733548    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container"
Jul 20 10:29:21.733625 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.733553    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container"
Jul 20 10:29:21.733625 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.733591    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kserve-container"
Jul 20 10:29:21.733625 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.733600    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" containerName="kube-rbac-proxy"
Jul 20 10:29:21.737622 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.737603    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.740372 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.740349    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:29:21.740598 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.740582    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:29:21.740674 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.740614    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:29:21.740674 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.740655    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:29:21.740773 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.740670    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 20 10:29:21.740773 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.740657    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 20 10:29:21.741384 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.741366    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:29:21.747002 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.746981    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"]
Jul 20 10:29:21.878293 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.878261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hgxg\" (UniqueName: \"kubernetes.io/projected/956179d2-5273-4b1f-a815-198ca261379d-kube-api-access-4hgxg\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.878661 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.878321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/956179d2-5273-4b1f-a815-198ca261379d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.878661 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.878367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/956179d2-5273-4b1f-a815-198ca261379d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.878661 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.878386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/956179d2-5273-4b1f-a815-198ca261379d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.878661 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.878404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956179d2-5273-4b1f-a815-198ca261379d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.979550 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.979525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hgxg\" (UniqueName: \"kubernetes.io/projected/956179d2-5273-4b1f-a815-198ca261379d-kube-api-access-4hgxg\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.979689 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.979573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/956179d2-5273-4b1f-a815-198ca261379d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.979689 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.979598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/956179d2-5273-4b1f-a815-198ca261379d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.979689 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.979620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/956179d2-5273-4b1f-a815-198ca261379d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.979689 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.979638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956179d2-5273-4b1f-a815-198ca261379d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.979965 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:29:21.979730    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 20 10:29:21.979965 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:29:21.979813    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/956179d2-5273-4b1f-a815-198ca261379d-proxy-tls podName:956179d2-5273-4b1f-a815-198ca261379d nodeName:}" failed. No retries permitted until 2026-07-20 10:29:22.479793269 +0000 UTC m=+1942.317891832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/956179d2-5273-4b1f-a815-198ca261379d-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd" (UID: "956179d2-5273-4b1f-a815-198ca261379d") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 20 10:29:21.980077 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.980058    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/956179d2-5273-4b1f-a815-198ca261379d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.980371 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.980350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/956179d2-5273-4b1f-a815-198ca261379d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.980452 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.980388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/956179d2-5273-4b1f-a815-198ca261379d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:21.988511 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:21.988456    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hgxg\" (UniqueName: \"kubernetes.io/projected/956179d2-5273-4b1f-a815-198ca261379d-kube-api-access-4hgxg\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:22.483297 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:22.483252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956179d2-5273-4b1f-a815-198ca261379d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:22.485566 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:22.485537    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956179d2-5273-4b1f-a815-198ca261379d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:22.649229 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:22.649199    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:29:22.765305 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:22.765233    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="590e3a12-757b-4f88-aaaf-38d16d927996" path="/var/lib/kubelet/pods/590e3a12-757b-4f88-aaaf-38d16d927996/volumes"
Jul 20 10:29:22.776220 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:22.776196    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"]
Jul 20 10:29:22.778550 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:29:22.778524    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956179d2_5273_4b1f_a815_198ca261379d.slice/crio-1a309ad03771c05e359c517d9f3603d17429b6a4af31b04b335ea57f29fc7dc4 WatchSource:0}: Error finding container 1a309ad03771c05e359c517d9f3603d17429b6a4af31b04b335ea57f29fc7dc4: Status 404 returned error can't find the container with id 1a309ad03771c05e359c517d9f3603d17429b6a4af31b04b335ea57f29fc7dc4
Jul 20 10:29:22.894209 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:22.894175    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd" event={"ID":"956179d2-5273-4b1f-a815-198ca261379d","Type":"ContainerStarted","Data":"c764d687a076fafa829adc42353f68de525ec25072869bb34ca277bd7ba190dd"}
Jul 20 10:29:22.894569 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:22.894238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd" event={"ID":"956179d2-5273-4b1f-a815-198ca261379d","Type":"ContainerStarted","Data":"1a309ad03771c05e359c517d9f3603d17429b6a4af31b04b335ea57f29fc7dc4"}
Jul 20 10:29:26.909739 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:26.909711    2561 generic.go:356] "Generic (PLEG): container finished" podID="956179d2-5273-4b1f-a815-198ca261379d" containerID="c764d687a076fafa829adc42353f68de525ec25072869bb34ca277bd7ba190dd" exitCode=0
Jul 20 10:29:26.910054 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:29:26.909752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd" event={"ID":"956179d2-5273-4b1f-a815-198ca261379d","Type":"ContainerDied","Data":"c764d687a076fafa829adc42353f68de525ec25072869bb34ca277bd7ba190dd"}
Jul 20 10:31:36.450069 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:31:36.450035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd" event={"ID":"956179d2-5273-4b1f-a815-198ca261379d","Type":"ContainerStarted","Data":"80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f"}
Jul 20 10:31:36.450547 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:31:36.450090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd" event={"ID":"956179d2-5273-4b1f-a815-198ca261379d","Type":"ContainerStarted","Data":"e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c"}
Jul 20 10:31:36.450547 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:31:36.450228    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:31:36.483233 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:31:36.483175    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd" podStartSLOduration=3.023918751 podStartE2EDuration="2m15.483159077s" totalImagesPullingTime="2m8.531859311s" totalInitContainerRuntime="3.927381015s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:31:36.482929774 +0000 UTC m=+2076.321028380" watchObservedRunningTime="2026-07-20 10:31:36.483159077 +0000 UTC m=+2076.321257662"
Jul 20 10:31:37.454969 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:31:37.454938    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:31:43.465388 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:31:43.465359    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:32:13.469077 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:13.469046    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:32:16.076201 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:16.076168    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:32:16.078562 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:16.078545    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:32:21.927598 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:21.927567    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"]
Jul 20 10:32:21.928151 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:21.927809    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd" podUID="956179d2-5273-4b1f-a815-198ca261379d" containerName="kserve-container" containerID="cri-o://e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c" gracePeriod=30
Jul 20 10:32:21.928151 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:21.927831    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd" podUID="956179d2-5273-4b1f-a815-198ca261379d" containerName="kube-rbac-proxy" containerID="cri-o://80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f" gracePeriod=30
Jul 20 10:32:22.617075 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.617038    2561 generic.go:356] "Generic (PLEG): container finished" podID="956179d2-5273-4b1f-a815-198ca261379d" containerID="80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f" exitCode=2
Jul 20 10:32:22.617075 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.617069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd" event={"ID":"956179d2-5273-4b1f-a815-198ca261379d","Type":"ContainerDied","Data":"80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f"}
Jul 20 10:32:22.869248 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.869190    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:32:22.987399 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.987365    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hgxg\" (UniqueName: \"kubernetes.io/projected/956179d2-5273-4b1f-a815-198ca261379d-kube-api-access-4hgxg\") pod \"956179d2-5273-4b1f-a815-198ca261379d\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") "
Jul 20 10:32:22.987870 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.987412    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/956179d2-5273-4b1f-a815-198ca261379d-kserve-provision-location\") pod \"956179d2-5273-4b1f-a815-198ca261379d\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") "
Jul 20 10:32:22.987870 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.987441    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/956179d2-5273-4b1f-a815-198ca261379d-kube-rbac-proxy-sar-config\") pod \"956179d2-5273-4b1f-a815-198ca261379d\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") "
Jul 20 10:32:22.987870 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.987469    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956179d2-5273-4b1f-a815-198ca261379d-proxy-tls\") pod \"956179d2-5273-4b1f-a815-198ca261379d\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") "
Jul 20 10:32:22.987870 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.987537    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/956179d2-5273-4b1f-a815-198ca261379d-cabundle-cert\") pod \"956179d2-5273-4b1f-a815-198ca261379d\" (UID: \"956179d2-5273-4b1f-a815-198ca261379d\") "
Jul 20 10:32:22.987870 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.987787    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956179d2-5273-4b1f-a815-198ca261379d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "956179d2-5273-4b1f-a815-198ca261379d" (UID: "956179d2-5273-4b1f-a815-198ca261379d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:32:22.987870 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.987824    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956179d2-5273-4b1f-a815-198ca261379d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "956179d2-5273-4b1f-a815-198ca261379d" (UID: "956179d2-5273-4b1f-a815-198ca261379d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:32:22.988103 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.987882    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956179d2-5273-4b1f-a815-198ca261379d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "956179d2-5273-4b1f-a815-198ca261379d" (UID: "956179d2-5273-4b1f-a815-198ca261379d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:32:22.989543 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.989524    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956179d2-5273-4b1f-a815-198ca261379d-kube-api-access-4hgxg" (OuterVolumeSpecName: "kube-api-access-4hgxg") pod "956179d2-5273-4b1f-a815-198ca261379d" (UID: "956179d2-5273-4b1f-a815-198ca261379d"). InnerVolumeSpecName "kube-api-access-4hgxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:32:22.989603 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:22.989571    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/956179d2-5273-4b1f-a815-198ca261379d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "956179d2-5273-4b1f-a815-198ca261379d" (UID: "956179d2-5273-4b1f-a815-198ca261379d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:32:23.088206 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.088179    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/956179d2-5273-4b1f-a815-198ca261379d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:32:23.088206 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.088203    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956179d2-5273-4b1f-a815-198ca261379d-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:32:23.088206 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.088212    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/956179d2-5273-4b1f-a815-198ca261379d-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:32:23.088431 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.088220    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hgxg\" (UniqueName: \"kubernetes.io/projected/956179d2-5273-4b1f-a815-198ca261379d-kube-api-access-4hgxg\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:32:23.088431 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.088230    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/956179d2-5273-4b1f-a815-198ca261379d-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:32:23.621967 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.621939    2561 generic.go:356] "Generic (PLEG): container finished" podID="956179d2-5273-4b1f-a815-198ca261379d" containerID="e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c" exitCode=0
Jul 20 10:32:23.622140 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.621990    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"
Jul 20 10:32:23.622140 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.622029    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd" event={"ID":"956179d2-5273-4b1f-a815-198ca261379d","Type":"ContainerDied","Data":"e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c"}
Jul 20 10:32:23.622140 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.622100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd" event={"ID":"956179d2-5273-4b1f-a815-198ca261379d","Type":"ContainerDied","Data":"1a309ad03771c05e359c517d9f3603d17429b6a4af31b04b335ea57f29fc7dc4"}
Jul 20 10:32:23.622341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.622157    2561 scope.go:117] "RemoveContainer" containerID="80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f"
Jul 20 10:32:23.630527 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.630512    2561 scope.go:117] "RemoveContainer" containerID="e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c"
Jul 20 10:32:23.637570 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.637554    2561 scope.go:117] "RemoveContainer" containerID="c764d687a076fafa829adc42353f68de525ec25072869bb34ca277bd7ba190dd"
Jul 20 10:32:23.644687 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.644656    2561 scope.go:117] "RemoveContainer" containerID="80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f"
Jul 20 10:32:23.644971 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:32:23.644953    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f\": container with ID starting with 80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f not found: ID does not exist" containerID="80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f"
Jul 20 10:32:23.645029 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.644980    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f"} err="failed to get container status \"80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f\": rpc error: code = NotFound desc = could not find container \"80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f\": container with ID starting with 80c9a6600c156a0318a64cf52357b62e585cab10f6c395a48ae19eac188f9c2f not found: ID does not exist"
Jul 20 10:32:23.645029 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.644997    2561 scope.go:117] "RemoveContainer" containerID="e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c"
Jul 20 10:32:23.645297 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:32:23.645279    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c\": container with ID starting with e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c not found: ID does not exist" containerID="e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c"
Jul 20 10:32:23.645374 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.645302    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c"} err="failed to get container status \"e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c\": rpc error: code = NotFound desc = could not find container \"e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c\": container with ID starting with e83a29a04e3b474bedf80d5c1279dac92dcc6d628bbe037aa05eeed8292ce97c not found: ID does not exist"
Jul 20 10:32:23.645374 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.645322    2561 scope.go:117] "RemoveContainer" containerID="c764d687a076fafa829adc42353f68de525ec25072869bb34ca277bd7ba190dd"
Jul 20 10:32:23.645746 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:32:23.645697    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c764d687a076fafa829adc42353f68de525ec25072869bb34ca277bd7ba190dd\": container with ID starting with c764d687a076fafa829adc42353f68de525ec25072869bb34ca277bd7ba190dd not found: ID does not exist" containerID="c764d687a076fafa829adc42353f68de525ec25072869bb34ca277bd7ba190dd"
Jul 20 10:32:23.645969 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.645750    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c764d687a076fafa829adc42353f68de525ec25072869bb34ca277bd7ba190dd"} err="failed to get container status \"c764d687a076fafa829adc42353f68de525ec25072869bb34ca277bd7ba190dd\": rpc error: code = NotFound desc = could not find container \"c764d687a076fafa829adc42353f68de525ec25072869bb34ca277bd7ba190dd\": container with ID starting with c764d687a076fafa829adc42353f68de525ec25072869bb34ca277bd7ba190dd not found: ID does not exist"
Jul 20 10:32:23.647061 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.647027    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"]
Jul 20 10:32:23.650335 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:23.650311    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gxjbd"]
Jul 20 10:32:24.765373 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:24.765332    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="956179d2-5273-4b1f-a815-198ca261379d" path="/var/lib/kubelet/pods/956179d2-5273-4b1f-a815-198ca261379d/volumes"
Jul 20 10:32:27.118920 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.118884    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"]
Jul 20 10:32:27.119476 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.119140    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="956179d2-5273-4b1f-a815-198ca261379d" containerName="kube-rbac-proxy"
Jul 20 10:32:27.119476 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.119152    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="956179d2-5273-4b1f-a815-198ca261379d" containerName="kube-rbac-proxy"
Jul 20 10:32:27.119476 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.119164    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="956179d2-5273-4b1f-a815-198ca261379d" containerName="kserve-container"
Jul 20 10:32:27.119476 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.119170    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="956179d2-5273-4b1f-a815-198ca261379d" containerName="kserve-container"
Jul 20 10:32:27.119476 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.119180    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="956179d2-5273-4b1f-a815-198ca261379d" containerName="storage-initializer"
Jul 20 10:32:27.119476 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.119186    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="956179d2-5273-4b1f-a815-198ca261379d" containerName="storage-initializer"
Jul 20 10:32:27.119476 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.119231    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="956179d2-5273-4b1f-a815-198ca261379d" containerName="kserve-container"
Jul 20 10:32:27.119476 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.119237    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="956179d2-5273-4b1f-a815-198ca261379d" containerName="kube-rbac-proxy"
Jul 20 10:32:27.122344 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.122325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.128121 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.128085    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 20 10:32:27.128663 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.128473    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:32:27.128663 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.128596    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:32:27.129000 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.128979    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:32:27.129222 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.129201    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 20 10:32:27.129324 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.129272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:32:27.129743 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.129729    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:32:27.139893 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.139869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"]
Jul 20 10:32:27.213778 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.213742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbac086d-b331-46d2-9a8b-b9844c282f4d-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.214000 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.213800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbac086d-b331-46d2-9a8b-b9844c282f4d-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.214000 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.213856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5d2b\" (UniqueName: \"kubernetes.io/projected/dbac086d-b331-46d2-9a8b-b9844c282f4d-kube-api-access-x5d2b\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.214000 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.213895    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbac086d-b331-46d2-9a8b-b9844c282f4d-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.214000 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.213949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbac086d-b331-46d2-9a8b-b9844c282f4d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.314565 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.314539    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbac086d-b331-46d2-9a8b-b9844c282f4d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.314723 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.314571    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbac086d-b331-46d2-9a8b-b9844c282f4d-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.314723 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.314598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbac086d-b331-46d2-9a8b-b9844c282f4d-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.314723 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:32:27.314704    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Jul 20 10:32:27.314723 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.314714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5d2b\" (UniqueName: \"kubernetes.io/projected/dbac086d-b331-46d2-9a8b-b9844c282f4d-kube-api-access-x5d2b\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.314933 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.314759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbac086d-b331-46d2-9a8b-b9844c282f4d-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.314933 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:32:27.314785    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbac086d-b331-46d2-9a8b-b9844c282f4d-proxy-tls podName:dbac086d-b331-46d2-9a8b-b9844c282f4d nodeName:}" failed. No retries permitted until 2026-07-20 10:32:27.814762672 +0000 UTC m=+2127.652861241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dbac086d-b331-46d2-9a8b-b9844c282f4d-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" (UID: "dbac086d-b331-46d2-9a8b-b9844c282f4d") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Jul 20 10:32:27.315144 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.315098    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbac086d-b331-46d2-9a8b-b9844c282f4d-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.315367 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.315351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbac086d-b331-46d2-9a8b-b9844c282f4d-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.315403 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.315383    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbac086d-b331-46d2-9a8b-b9844c282f4d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.324522 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.324492    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5d2b\" (UniqueName: \"kubernetes.io/projected/dbac086d-b331-46d2-9a8b-b9844c282f4d-kube-api-access-x5d2b\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.817364 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.817319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbac086d-b331-46d2-9a8b-b9844c282f4d-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:27.819709 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:27.819678    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbac086d-b331-46d2-9a8b-b9844c282f4d-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:28.032685 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:28.032648    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:28.158373 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:28.158331    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"]
Jul 20 10:32:28.160693 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:32:28.160659    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbac086d_b331_46d2_9a8b_b9844c282f4d.slice/crio-98b79fa676f052817ddca9ad0459c7c2e8b077af74183bdebe508a7c8fd2bc39 WatchSource:0}: Error finding container 98b79fa676f052817ddca9ad0459c7c2e8b077af74183bdebe508a7c8fd2bc39: Status 404 returned error can't find the container with id 98b79fa676f052817ddca9ad0459c7c2e8b077af74183bdebe508a7c8fd2bc39
Jul 20 10:32:28.642725 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:28.642691    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" event={"ID":"dbac086d-b331-46d2-9a8b-b9844c282f4d","Type":"ContainerStarted","Data":"15217e488e08aef832e928618071ed2c28b62f5f75912b6fcdf8012c5b392777"}
Jul 20 10:32:28.642886 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:28.642755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" event={"ID":"dbac086d-b331-46d2-9a8b-b9844c282f4d","Type":"ContainerStarted","Data":"98b79fa676f052817ddca9ad0459c7c2e8b077af74183bdebe508a7c8fd2bc39"}
Jul 20 10:32:32.658272 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:32.658237    2561 generic.go:356] "Generic (PLEG): container finished" podID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerID="15217e488e08aef832e928618071ed2c28b62f5f75912b6fcdf8012c5b392777" exitCode=0
Jul 20 10:32:32.658272 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:32.658274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" event={"ID":"dbac086d-b331-46d2-9a8b-b9844c282f4d","Type":"ContainerDied","Data":"15217e488e08aef832e928618071ed2c28b62f5f75912b6fcdf8012c5b392777"}
Jul 20 10:32:33.664926 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:33.664888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" event={"ID":"dbac086d-b331-46d2-9a8b-b9844c282f4d","Type":"ContainerStarted","Data":"9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4"}
Jul 20 10:32:33.665413 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:33.664965    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" event={"ID":"dbac086d-b331-46d2-9a8b-b9844c282f4d","Type":"ContainerStarted","Data":"7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9"}
Jul 20 10:32:33.685007 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:33.684952    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" podStartSLOduration=2.864752225 podStartE2EDuration="6.684933299s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.820181074s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:32:33.684315539 +0000 UTC m=+2133.522414181" watchObservedRunningTime="2026-07-20 10:32:33.684933299 +0000 UTC m=+2133.523031887"
Jul 20 10:32:38.666170 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:38.666109    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:38.666695 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:38.666670    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:38.667434 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:38.667407    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 20 10:32:38.670808 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:38.670785    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:38.684172 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:38.684141    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 20 10:32:39.688097 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:39.688062    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:47.227818 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.227788    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"]
Jul 20 10:32:47.228293 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.228023    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerName="kserve-container" containerID="cri-o://7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9" gracePeriod=30
Jul 20 10:32:47.228293 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.228076    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerName="kube-rbac-proxy" containerID="cri-o://9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4" gracePeriod=30
Jul 20 10:32:47.718511 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.718483    2561 generic.go:356] "Generic (PLEG): container finished" podID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerID="9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4" exitCode=2
Jul 20 10:32:47.718618 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.718529    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" event={"ID":"dbac086d-b331-46d2-9a8b-b9844c282f4d","Type":"ContainerDied","Data":"9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4"}
Jul 20 10:32:47.862746 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.862722    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:47.972498 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.972425    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbac086d-b331-46d2-9a8b-b9844c282f4d-proxy-tls\") pod \"dbac086d-b331-46d2-9a8b-b9844c282f4d\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") "
Jul 20 10:32:47.972498 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.972456    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbac086d-b331-46d2-9a8b-b9844c282f4d-kserve-provision-location\") pod \"dbac086d-b331-46d2-9a8b-b9844c282f4d\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") "
Jul 20 10:32:47.972498 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.972483    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbac086d-b331-46d2-9a8b-b9844c282f4d-cabundle-cert\") pod \"dbac086d-b331-46d2-9a8b-b9844c282f4d\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") "
Jul 20 10:32:47.972795 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.972519    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5d2b\" (UniqueName: \"kubernetes.io/projected/dbac086d-b331-46d2-9a8b-b9844c282f4d-kube-api-access-x5d2b\") pod \"dbac086d-b331-46d2-9a8b-b9844c282f4d\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") "
Jul 20 10:32:47.972795 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.972545    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbac086d-b331-46d2-9a8b-b9844c282f4d-kube-rbac-proxy-sar-config\") pod \"dbac086d-b331-46d2-9a8b-b9844c282f4d\" (UID: \"dbac086d-b331-46d2-9a8b-b9844c282f4d\") "
Jul 20 10:32:47.972902 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.972788    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbac086d-b331-46d2-9a8b-b9844c282f4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbac086d-b331-46d2-9a8b-b9844c282f4d" (UID: "dbac086d-b331-46d2-9a8b-b9844c282f4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:32:47.972948 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.972912    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbac086d-b331-46d2-9a8b-b9844c282f4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dbac086d-b331-46d2-9a8b-b9844c282f4d" (UID: "dbac086d-b331-46d2-9a8b-b9844c282f4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:32:47.972991 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.972961    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbac086d-b331-46d2-9a8b-b9844c282f4d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dbac086d-b331-46d2-9a8b-b9844c282f4d" (UID: "dbac086d-b331-46d2-9a8b-b9844c282f4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:32:47.974349 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.974328    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbac086d-b331-46d2-9a8b-b9844c282f4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbac086d-b331-46d2-9a8b-b9844c282f4d" (UID: "dbac086d-b331-46d2-9a8b-b9844c282f4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:32:47.974575 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:47.974558    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbac086d-b331-46d2-9a8b-b9844c282f4d-kube-api-access-x5d2b" (OuterVolumeSpecName: "kube-api-access-x5d2b") pod "dbac086d-b331-46d2-9a8b-b9844c282f4d" (UID: "dbac086d-b331-46d2-9a8b-b9844c282f4d"). InnerVolumeSpecName "kube-api-access-x5d2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:32:48.073916 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.073893    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbac086d-b331-46d2-9a8b-b9844c282f4d-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:32:48.073916 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.073914    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5d2b\" (UniqueName: \"kubernetes.io/projected/dbac086d-b331-46d2-9a8b-b9844c282f4d-kube-api-access-x5d2b\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:32:48.074049 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.073924    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbac086d-b331-46d2-9a8b-b9844c282f4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:32:48.074049 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.073934    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbac086d-b331-46d2-9a8b-b9844c282f4d-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:32:48.074049 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.073949    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbac086d-b331-46d2-9a8b-b9844c282f4d-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:32:48.724322 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.724286    2561 generic.go:356] "Generic (PLEG): container finished" podID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerID="7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9" exitCode=0
Jul 20 10:32:48.724717 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.724329    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" event={"ID":"dbac086d-b331-46d2-9a8b-b9844c282f4d","Type":"ContainerDied","Data":"7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9"}
Jul 20 10:32:48.724717 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.724351    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"
Jul 20 10:32:48.724717 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.724390    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr" event={"ID":"dbac086d-b331-46d2-9a8b-b9844c282f4d","Type":"ContainerDied","Data":"98b79fa676f052817ddca9ad0459c7c2e8b077af74183bdebe508a7c8fd2bc39"}
Jul 20 10:32:48.724717 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.724429    2561 scope.go:117] "RemoveContainer" containerID="9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4"
Jul 20 10:32:48.732950 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.732932    2561 scope.go:117] "RemoveContainer" containerID="7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9"
Jul 20 10:32:48.740394 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.740377    2561 scope.go:117] "RemoveContainer" containerID="15217e488e08aef832e928618071ed2c28b62f5f75912b6fcdf8012c5b392777"
Jul 20 10:32:48.748068 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.748035    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"]
Jul 20 10:32:48.748156 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.748090    2561 scope.go:117] "RemoveContainer" containerID="9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4"
Jul 20 10:32:48.748612 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:32:48.748593    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4\": container with ID starting with 9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4 not found: ID does not exist" containerID="9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4"
Jul 20 10:32:48.748683 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.748617    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4"} err="failed to get container status \"9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4\": rpc error: code = NotFound desc = could not find container \"9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4\": container with ID starting with 9053548ef09241692f6ff25e49eb01a5201deb4705fcabb9bc40dbc9679fdbd4 not found: ID does not exist"
Jul 20 10:32:48.748683 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.748635    2561 scope.go:117] "RemoveContainer" containerID="7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9"
Jul 20 10:32:48.748882 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:32:48.748867    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9\": container with ID starting with 7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9 not found: ID does not exist" containerID="7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9"
Jul 20 10:32:48.748946 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.748885    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9"} err="failed to get container status \"7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9\": rpc error: code = NotFound desc = could not find container \"7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9\": container with ID starting with 7aa07c7334540e1e6894f245a06cd991a0f7cac0594ba4410b74d09f9ef9eff9 not found: ID does not exist"
Jul 20 10:32:48.748946 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.748898    2561 scope.go:117] "RemoveContainer" containerID="15217e488e08aef832e928618071ed2c28b62f5f75912b6fcdf8012c5b392777"
Jul 20 10:32:48.749137 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:32:48.749103    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15217e488e08aef832e928618071ed2c28b62f5f75912b6fcdf8012c5b392777\": container with ID starting with 15217e488e08aef832e928618071ed2c28b62f5f75912b6fcdf8012c5b392777 not found: ID does not exist" containerID="15217e488e08aef832e928618071ed2c28b62f5f75912b6fcdf8012c5b392777"
Jul 20 10:32:48.749189 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.749141    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15217e488e08aef832e928618071ed2c28b62f5f75912b6fcdf8012c5b392777"} err="failed to get container status \"15217e488e08aef832e928618071ed2c28b62f5f75912b6fcdf8012c5b392777\": rpc error: code = NotFound desc = could not find container \"15217e488e08aef832e928618071ed2c28b62f5f75912b6fcdf8012c5b392777\": container with ID starting with 15217e488e08aef832e928618071ed2c28b62f5f75912b6fcdf8012c5b392777 not found: ID does not exist"
Jul 20 10:32:48.754082 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.754062    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-thzrr"]
Jul 20 10:32:48.764926 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:48.764905    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" path="/var/lib/kubelet/pods/dbac086d-b331-46d2-9a8b-b9844c282f4d/volumes"
Jul 20 10:32:52.313406 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.313371    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"]
Jul 20 10:32:52.314747 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.313663    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerName="kserve-container"
Jul 20 10:32:52.314747 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.313677    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerName="kserve-container"
Jul 20 10:32:52.314747 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.313684    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerName="kube-rbac-proxy"
Jul 20 10:32:52.314747 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.313690    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerName="kube-rbac-proxy"
Jul 20 10:32:52.314747 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.313697    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerName="storage-initializer"
Jul 20 10:32:52.314747 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.313702    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerName="storage-initializer"
Jul 20 10:32:52.314747 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.313752    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerName="kube-rbac-proxy"
Jul 20 10:32:52.314747 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.313761    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbac086d-b331-46d2-9a8b-b9844c282f4d" containerName="kserve-container"
Jul 20 10:32:52.315411 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.315396    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.320765 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.320735    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 20 10:32:52.321003 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.320841    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:32:52.321729 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.321318    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:32:52.321729 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.321371    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:32:52.321729 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.321324    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 20 10:32:52.321729 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.321324    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:32:52.321978 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.321824    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:32:52.334384 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.334359    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"]
Jul 20 10:32:52.406124 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.406090    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.406289 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.406174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.406289 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.406207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.406289 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.406238    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.406448 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.406293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwf2v\" (UniqueName: \"kubernetes.io/projected/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kube-api-access-rwf2v\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.507303 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.507272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwf2v\" (UniqueName: \"kubernetes.io/projected/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kube-api-access-rwf2v\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.507435 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.507317    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.507435 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.507370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.507435 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.507405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.507544 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.507525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.507955 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.507932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.508105 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.508087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.508186 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.508152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.509882 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.509864    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.536057 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.536036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwf2v\" (UniqueName: \"kubernetes.io/projected/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kube-api-access-rwf2v\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.625795 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.625769    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:52.752015 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:52.751953    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"]
Jul 20 10:32:52.754126 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:32:52.754078    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4165c8_0a2f_4077_b1bd_e4cdfad1f71b.slice/crio-a10702743743e9da6dfcd51ca7efc2ebd0e73ab00522dc8aabd0398a2a7194f5 WatchSource:0}: Error finding container a10702743743e9da6dfcd51ca7efc2ebd0e73ab00522dc8aabd0398a2a7194f5: Status 404 returned error can't find the container with id a10702743743e9da6dfcd51ca7efc2ebd0e73ab00522dc8aabd0398a2a7194f5
Jul 20 10:32:53.748083 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:53.748046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p" event={"ID":"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b","Type":"ContainerStarted","Data":"619feada7ec30220041a3e8b2d036324b110191dbd7057180c3609ed058811c4"}
Jul 20 10:32:53.748557 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:53.748126    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p" event={"ID":"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b","Type":"ContainerStarted","Data":"a10702743743e9da6dfcd51ca7efc2ebd0e73ab00522dc8aabd0398a2a7194f5"}
Jul 20 10:32:56.759804 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:56.759771    2561 generic.go:356] "Generic (PLEG): container finished" podID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerID="619feada7ec30220041a3e8b2d036324b110191dbd7057180c3609ed058811c4" exitCode=0
Jul 20 10:32:56.760165 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:56.759844    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p" event={"ID":"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b","Type":"ContainerDied","Data":"619feada7ec30220041a3e8b2d036324b110191dbd7057180c3609ed058811c4"}
Jul 20 10:32:57.766146 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:57.766093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p" event={"ID":"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b","Type":"ContainerStarted","Data":"b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd"}
Jul 20 10:32:57.766547 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:57.766196    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p" event={"ID":"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b","Type":"ContainerStarted","Data":"c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4"}
Jul 20 10:32:57.766547 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:57.766429    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:57.766631 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:57.766591    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:32:57.787945 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:32:57.787899    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p" podStartSLOduration=2.006426661 podStartE2EDuration="5.787884817s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.781458156s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:32:57.787360907 +0000 UTC m=+2157.625459494" watchObservedRunningTime="2026-07-20 10:32:57.787884817 +0000 UTC m=+2157.625983401"
Jul 20 10:33:03.782249 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:03.782220    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:33:33.786019 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:33.785984    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:33:42.436454 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:42.436424    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"]
Jul 20 10:33:42.436820 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:42.436658    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p" podUID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerName="kserve-container" containerID="cri-o://c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4" gracePeriod=30
Jul 20 10:33:42.436820 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:42.436712    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p" podUID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerName="kube-rbac-proxy" containerID="cri-o://b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd" gracePeriod=30
Jul 20 10:33:42.935449 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:42.935414    2561 generic.go:356] "Generic (PLEG): container finished" podID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerID="b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd" exitCode=2
Jul 20 10:33:42.935604 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:42.935484    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p" event={"ID":"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b","Type":"ContainerDied","Data":"b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd"}
Jul 20 10:33:43.577236 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.577216    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:33:43.661073 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.661046    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-cabundle-cert\") pod \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") "
Jul 20 10:33:43.661202 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.661086    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kserve-provision-location\") pod \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") "
Jul 20 10:33:43.661202 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.661170    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-proxy-tls\") pod \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") "
Jul 20 10:33:43.661291 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.661225    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kube-rbac-proxy-sar-config\") pod \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") "
Jul 20 10:33:43.661341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.661288    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwf2v\" (UniqueName: \"kubernetes.io/projected/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kube-api-access-rwf2v\") pod \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\" (UID: \"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b\") "
Jul 20 10:33:43.661415 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.661392    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" (UID: "1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:33:43.661477 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.661414    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" (UID: "1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:33:43.661544 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.661503    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:33:43.661544 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.661521    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:33:43.661681 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.661657    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" (UID: "1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:33:43.663203 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.663183    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" (UID: "1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:33:43.663313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.663294    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kube-api-access-rwf2v" (OuterVolumeSpecName: "kube-api-access-rwf2v") pod "1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" (UID: "1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b"). InnerVolumeSpecName "kube-api-access-rwf2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:33:43.762031 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.762004    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:33:43.762031 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.762026    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwf2v\" (UniqueName: \"kubernetes.io/projected/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-kube-api-access-rwf2v\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:33:43.762204 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.762036    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:33:43.940780 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.940705    2561 generic.go:356] "Generic (PLEG): container finished" podID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerID="c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4" exitCode=0
Jul 20 10:33:43.940780 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.940738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p" event={"ID":"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b","Type":"ContainerDied","Data":"c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4"}
Jul 20 10:33:43.940780 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.940771    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"
Jul 20 10:33:43.941022 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.940789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p" event={"ID":"1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b","Type":"ContainerDied","Data":"a10702743743e9da6dfcd51ca7efc2ebd0e73ab00522dc8aabd0398a2a7194f5"}
Jul 20 10:33:43.941022 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.940817    2561 scope.go:117] "RemoveContainer" containerID="b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd"
Jul 20 10:33:43.949005 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.948986    2561 scope.go:117] "RemoveContainer" containerID="c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4"
Jul 20 10:33:43.956430 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.956413    2561 scope.go:117] "RemoveContainer" containerID="619feada7ec30220041a3e8b2d036324b110191dbd7057180c3609ed058811c4"
Jul 20 10:33:43.963534 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.963517    2561 scope.go:117] "RemoveContainer" containerID="b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd"
Jul 20 10:33:43.963840 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:33:43.963807    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd\": container with ID starting with b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd not found: ID does not exist" containerID="b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd"
Jul 20 10:33:43.963908 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.963835    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd"} err="failed to get container status \"b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd\": rpc error: code = NotFound desc = could not find container \"b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd\": container with ID starting with b90a455ba3d258c516b7a01b715fe057f291bed55fdbf3960dab8934b90b87bd not found: ID does not exist"
Jul 20 10:33:43.963908 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.963853    2561 scope.go:117] "RemoveContainer" containerID="c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4"
Jul 20 10:33:43.964069 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:33:43.964054    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4\": container with ID starting with c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4 not found: ID does not exist" containerID="c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4"
Jul 20 10:33:43.964127 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.964073    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4"} err="failed to get container status \"c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4\": rpc error: code = NotFound desc = could not find container \"c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4\": container with ID starting with c24b53f4bb02b74330afbfba7a2927efb3292880bf739edb576f1f2fcce4d1f4 not found: ID does not exist"
Jul 20 10:33:43.964127 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.964086    2561 scope.go:117] "RemoveContainer" containerID="619feada7ec30220041a3e8b2d036324b110191dbd7057180c3609ed058811c4"
Jul 20 10:33:43.964347 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:33:43.964329    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619feada7ec30220041a3e8b2d036324b110191dbd7057180c3609ed058811c4\": container with ID starting with 619feada7ec30220041a3e8b2d036324b110191dbd7057180c3609ed058811c4 not found: ID does not exist" containerID="619feada7ec30220041a3e8b2d036324b110191dbd7057180c3609ed058811c4"
Jul 20 10:33:43.964405 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.964355    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619feada7ec30220041a3e8b2d036324b110191dbd7057180c3609ed058811c4"} err="failed to get container status \"619feada7ec30220041a3e8b2d036324b110191dbd7057180c3609ed058811c4\": rpc error: code = NotFound desc = could not find container \"619feada7ec30220041a3e8b2d036324b110191dbd7057180c3609ed058811c4\": container with ID starting with 619feada7ec30220041a3e8b2d036324b110191dbd7057180c3609ed058811c4 not found: ID does not exist"
Jul 20 10:33:43.964447 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.964430    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"]
Jul 20 10:33:43.968333 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:43.968315    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-f7x8p"]
Jul 20 10:33:44.766211 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:44.766177    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" path="/var/lib/kubelet/pods/1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b/volumes"
Jul 20 10:33:47.608294 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.608261    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"]
Jul 20 10:33:47.608666 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.608505    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerName="storage-initializer"
Jul 20 10:33:47.608666 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.608515    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerName="storage-initializer"
Jul 20 10:33:47.608666 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.608525    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerName="kserve-container"
Jul 20 10:33:47.608666 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.608530    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerName="kserve-container"
Jul 20 10:33:47.608666 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.608549    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerName="kube-rbac-proxy"
Jul 20 10:33:47.608666 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.608555    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerName="kube-rbac-proxy"
Jul 20 10:33:47.608666 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.608597    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerName="kube-rbac-proxy"
Jul 20 10:33:47.608666 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.608606    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f4165c8-0a2f-4077-b1bd-e4cdfad1f71b" containerName="kserve-container"
Jul 20 10:33:47.611551 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.611530    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.614327 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.614310    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:33:47.614467 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.614450    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:33:47.614581 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.614557    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 20 10:33:47.614661 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.614621    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:33:47.614857 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.614843    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:33:47.615555 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.615534    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 20 10:33:47.615555 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.615552    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:33:47.624359 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.624335    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"]
Jul 20 10:33:47.691038 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.691016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c9dd00-01dd-402b-a437-05e70e7a82a8-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.691183 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.691053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n2kk\" (UniqueName: \"kubernetes.io/projected/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kube-api-access-6n2kk\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.691183 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.691100    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c9dd00-01dd-402b-a437-05e70e7a82a8-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.691183 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.691162    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.691309 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.691192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.792375 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.792336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c9dd00-01dd-402b-a437-05e70e7a82a8-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.792375 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.792377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.792542 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.792395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.792542 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.792421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c9dd00-01dd-402b-a437-05e70e7a82a8-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.792542 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.792462    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2kk\" (UniqueName: \"kubernetes.io/projected/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kube-api-access-6n2kk\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.792784 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.792768    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.793159 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.793103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.793267 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.793244    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c9dd00-01dd-402b-a437-05e70e7a82a8-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.794893 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.794875    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c9dd00-01dd-402b-a437-05e70e7a82a8-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.800973 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.800948    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n2kk\" (UniqueName: \"kubernetes.io/projected/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kube-api-access-6n2kk\") pod \"isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:47.921704 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:47.921684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:48.058082 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:48.058056    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"]
Jul 20 10:33:48.060382 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:33:48.060354    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c9dd00_01dd_402b_a437_05e70e7a82a8.slice/crio-5281542b8b57d6d5b7c5ec4ab83d75355f6399e9fd3037f199e781eaa8898756 WatchSource:0}: Error finding container 5281542b8b57d6d5b7c5ec4ab83d75355f6399e9fd3037f199e781eaa8898756: Status 404 returned error can't find the container with id 5281542b8b57d6d5b7c5ec4ab83d75355f6399e9fd3037f199e781eaa8898756
Jul 20 10:33:48.962076 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:48.962045    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" event={"ID":"d0c9dd00-01dd-402b-a437-05e70e7a82a8","Type":"ContainerStarted","Data":"96f25bf86658769edc0af59f660f1e8f6ba88b71e8df36636e14c99c75924eb6"}
Jul 20 10:33:48.962477 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:48.962109    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" event={"ID":"d0c9dd00-01dd-402b-a437-05e70e7a82a8","Type":"ContainerStarted","Data":"5281542b8b57d6d5b7c5ec4ab83d75355f6399e9fd3037f199e781eaa8898756"}
Jul 20 10:33:51.973857 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:51.973824    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerID="96f25bf86658769edc0af59f660f1e8f6ba88b71e8df36636e14c99c75924eb6" exitCode=0
Jul 20 10:33:51.974218 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:51.973899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" event={"ID":"d0c9dd00-01dd-402b-a437-05e70e7a82a8","Type":"ContainerDied","Data":"96f25bf86658769edc0af59f660f1e8f6ba88b71e8df36636e14c99c75924eb6"}
Jul 20 10:33:52.024683 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:52.024664    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:33:52.980157 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:52.980100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" event={"ID":"d0c9dd00-01dd-402b-a437-05e70e7a82a8","Type":"ContainerStarted","Data":"789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1"}
Jul 20 10:33:54.990226 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:54.990185    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" event={"ID":"d0c9dd00-01dd-402b-a437-05e70e7a82a8","Type":"ContainerStarted","Data":"cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb"}
Jul 20 10:33:54.990588 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:54.990252    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" event={"ID":"d0c9dd00-01dd-402b-a437-05e70e7a82a8","Type":"ContainerStarted","Data":"0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52"}
Jul 20 10:33:54.990588 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:54.990281    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:54.990588 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:54.990308    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:33:55.014629 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:55.014580    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podStartSLOduration=2.321930028 podStartE2EDuration="8.014566003s" totalImagesPullingTime="2.026403757s" totalInitContainerRuntime="3.666232218s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:33:55.01351833 +0000 UTC m=+2214.851616915" watchObservedRunningTime="2026-07-20 10:33:55.014566003 +0000 UTC m=+2214.852664584"
Jul 20 10:33:55.993293 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:33:55.993264    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:34:02.001878 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:34:02.001849    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:34:22.002773 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:34:22.002735    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 20 10:34:32.004550 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:34:32.004515    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:35:02.005188 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:02.005161    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:35:07.721952 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:07.721919    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"]
Jul 20 10:35:07.722527 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:07.722182    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kserve-container" containerID="cri-o://789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1" gracePeriod=30
Jul 20 10:35:07.722527 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:07.722222    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kserve-agent" containerID="cri-o://0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52" gracePeriod=30
Jul 20 10:35:07.722527 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:07.722235    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kube-rbac-proxy" containerID="cri-o://cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb" gracePeriod=30
Jul 20 10:35:08.263936 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:08.263903    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerID="cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb" exitCode=2
Jul 20 10:35:08.264141 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:08.263976    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" event={"ID":"d0c9dd00-01dd-402b-a437-05e70e7a82a8","Type":"ContainerDied","Data":"cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb"}
Jul 20 10:35:10.273989 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:10.273961    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerID="789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1" exitCode=0
Jul 20 10:35:10.274386 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:10.274013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" event={"ID":"d0c9dd00-01dd-402b-a437-05e70e7a82a8","Type":"ContainerDied","Data":"789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1"}
Jul 20 10:35:11.997378 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:11.997337    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 20 10:35:12.002789 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.002761    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 20 10:35:12.868813 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.868777    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"]
Jul 20 10:35:12.872006 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.871984    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:12.874468 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.874445    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 20 10:35:12.874583 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.874490    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 20 10:35:12.885267 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.885243    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"]
Jul 20 10:35:12.898245 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.898222    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/476ed2a3-72a1-4399-9322-4fd213a5c8bc-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:12.898344 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.898255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:12.898344 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.898295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:12.898434 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.898347    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf6jn\" (UniqueName: \"kubernetes.io/projected/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kube-api-access-hf6jn\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:12.898434 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.898390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/476ed2a3-72a1-4399-9322-4fd213a5c8bc-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:12.999472 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.999443    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:12.999831 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.999484    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf6jn\" (UniqueName: \"kubernetes.io/projected/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kube-api-access-hf6jn\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:12.999831 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.999503    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/476ed2a3-72a1-4399-9322-4fd213a5c8bc-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:12.999831 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.999538    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/476ed2a3-72a1-4399-9322-4fd213a5c8bc-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:12.999831 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:12.999559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:13.000060 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:13.000033    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:13.000102 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:13.000061    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:13.000211 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:13.000196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/476ed2a3-72a1-4399-9322-4fd213a5c8bc-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:13.001848 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:13.001829    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/476ed2a3-72a1-4399-9322-4fd213a5c8bc-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:13.009859 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:13.009835    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf6jn\" (UniqueName: \"kubernetes.io/projected/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kube-api-access-hf6jn\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b6h82\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:13.181989 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:13.181891    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:13.314647 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:13.314554    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"]
Jul 20 10:35:13.316808 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:35:13.316778    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476ed2a3_72a1_4399_9322_4fd213a5c8bc.slice/crio-6fcff64e5e8199f1d19a0c10059f7a111b9f4905b8c1782adb953f1c41cd7ab5 WatchSource:0}: Error finding container 6fcff64e5e8199f1d19a0c10059f7a111b9f4905b8c1782adb953f1c41cd7ab5: Status 404 returned error can't find the container with id 6fcff64e5e8199f1d19a0c10059f7a111b9f4905b8c1782adb953f1c41cd7ab5
Jul 20 10:35:14.292132 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:14.292078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" event={"ID":"476ed2a3-72a1-4399-9322-4fd213a5c8bc","Type":"ContainerStarted","Data":"172ec67fcc383319a09e47cc61c4ab6d1f84003b6792412e80c0f00efdc74502"}
Jul 20 10:35:14.292517 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:14.292162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" event={"ID":"476ed2a3-72a1-4399-9322-4fd213a5c8bc","Type":"ContainerStarted","Data":"6fcff64e5e8199f1d19a0c10059f7a111b9f4905b8c1782adb953f1c41cd7ab5"}
Jul 20 10:35:16.997539 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:16.997487    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 20 10:35:18.307829 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:18.307798    2561 generic.go:356] "Generic (PLEG): container finished" podID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerID="172ec67fcc383319a09e47cc61c4ab6d1f84003b6792412e80c0f00efdc74502" exitCode=0
Jul 20 10:35:18.308231 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:18.307841    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" event={"ID":"476ed2a3-72a1-4399-9322-4fd213a5c8bc","Type":"ContainerDied","Data":"172ec67fcc383319a09e47cc61c4ab6d1f84003b6792412e80c0f00efdc74502"}
Jul 20 10:35:21.997318 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:21.997275    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 20 10:35:21.997755 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:21.997415    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:35:22.002518 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:22.002482    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 20 10:35:26.997526 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:26.997468    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 20 10:35:30.362767 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:30.362734    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" event={"ID":"476ed2a3-72a1-4399-9322-4fd213a5c8bc","Type":"ContainerStarted","Data":"7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a"}
Jul 20 10:35:30.363168 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:30.362793    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" event={"ID":"476ed2a3-72a1-4399-9322-4fd213a5c8bc","Type":"ContainerStarted","Data":"40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d"}
Jul 20 10:35:30.363168 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:30.362970    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:30.397158 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:30.397088    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" podStartSLOduration=2.451567473 podStartE2EDuration="18.39707277s" totalImagesPullingTime="11.484225865s" totalInitContainerRuntime="4.461279432s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:35:30.396238982 +0000 UTC m=+2310.234337579" watchObservedRunningTime="2026-07-20 10:35:30.39707277 +0000 UTC m=+2310.235171354"
Jul 20 10:35:31.367672 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:31.367639    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:31.368872 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:31.368845    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 20 10:35:31.997964 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:31.997922    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 20 10:35:32.003393 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:32.003365    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 20 10:35:32.003491 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:32.003478    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:35:32.372598 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:32.372562    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 20 10:35:36.997455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:36.997413    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 20 10:35:37.378172 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.378134    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:35:37.378746 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.378717    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 20 10:35:37.902714 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.902689    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:35:37.994556 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.994489    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kube-rbac-proxy-sar-config\") pod \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") "
Jul 20 10:35:37.994556 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.994550    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c9dd00-01dd-402b-a437-05e70e7a82a8-cabundle-cert\") pod \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") "
Jul 20 10:35:37.994771 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.994583    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c9dd00-01dd-402b-a437-05e70e7a82a8-proxy-tls\") pod \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") "
Jul 20 10:35:37.994771 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.994622    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n2kk\" (UniqueName: \"kubernetes.io/projected/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kube-api-access-6n2kk\") pod \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") "
Jul 20 10:35:37.994879 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.994776    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kserve-provision-location\") pod \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\" (UID: \"d0c9dd00-01dd-402b-a437-05e70e7a82a8\") "
Jul 20 10:35:37.994929 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.994875    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0c9dd00-01dd-402b-a437-05e70e7a82a8" (UID: "d0c9dd00-01dd-402b-a437-05e70e7a82a8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:35:37.995002 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.994970    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c9dd00-01dd-402b-a437-05e70e7a82a8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0c9dd00-01dd-402b-a437-05e70e7a82a8" (UID: "d0c9dd00-01dd-402b-a437-05e70e7a82a8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:35:37.995090 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.995077    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c9dd00-01dd-402b-a437-05e70e7a82a8-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:35:37.995172 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.995083    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0c9dd00-01dd-402b-a437-05e70e7a82a8" (UID: "d0c9dd00-01dd-402b-a437-05e70e7a82a8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:35:37.995172 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.995097    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:35:37.996709 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.996682    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kube-api-access-6n2kk" (OuterVolumeSpecName: "kube-api-access-6n2kk") pod "d0c9dd00-01dd-402b-a437-05e70e7a82a8" (UID: "d0c9dd00-01dd-402b-a437-05e70e7a82a8"). InnerVolumeSpecName "kube-api-access-6n2kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:35:37.996814 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:37.996695    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c9dd00-01dd-402b-a437-05e70e7a82a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0c9dd00-01dd-402b-a437-05e70e7a82a8" (UID: "d0c9dd00-01dd-402b-a437-05e70e7a82a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:35:38.096019 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.095993    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c9dd00-01dd-402b-a437-05e70e7a82a8-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:35:38.096019 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.096017    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n2kk\" (UniqueName: \"kubernetes.io/projected/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kube-api-access-6n2kk\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:35:38.096408 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.096027    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c9dd00-01dd-402b-a437-05e70e7a82a8-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:35:38.395169 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.395140    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerID="0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52" exitCode=137
Jul 20 10:35:38.395343 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.395251    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"
Jul 20 10:35:38.395343 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.395248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" event={"ID":"d0c9dd00-01dd-402b-a437-05e70e7a82a8","Type":"ContainerDied","Data":"0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52"}
Jul 20 10:35:38.395460 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.395406    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b" event={"ID":"d0c9dd00-01dd-402b-a437-05e70e7a82a8","Type":"ContainerDied","Data":"5281542b8b57d6d5b7c5ec4ab83d75355f6399e9fd3037f199e781eaa8898756"}
Jul 20 10:35:38.395519 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.395463    2561 scope.go:117] "RemoveContainer" containerID="cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb"
Jul 20 10:35:38.403384 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.403367    2561 scope.go:117] "RemoveContainer" containerID="0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52"
Jul 20 10:35:38.410612 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.410596    2561 scope.go:117] "RemoveContainer" containerID="789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1"
Jul 20 10:35:38.417334 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.417287    2561 scope.go:117] "RemoveContainer" containerID="96f25bf86658769edc0af59f660f1e8f6ba88b71e8df36636e14c99c75924eb6"
Jul 20 10:35:38.419876 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.419854    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"]
Jul 20 10:35:38.424820 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.424804    2561 scope.go:117] "RemoveContainer" containerID="cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb"
Jul 20 10:35:38.425045 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:35:38.425020    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb\": container with ID starting with cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb not found: ID does not exist" containerID="cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb"
Jul 20 10:35:38.425224 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.425044    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb"} err="failed to get container status \"cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb\": rpc error: code = NotFound desc = could not find container \"cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb\": container with ID starting with cbb59d016a303a8276fa488c076551805ef763a1a0552b6b48517bbfc8ce4acb not found: ID does not exist"
Jul 20 10:35:38.425224 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.425068    2561 scope.go:117] "RemoveContainer" containerID="0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52"
Jul 20 10:35:38.425355 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:35:38.425336    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52\": container with ID starting with 0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52 not found: ID does not exist" containerID="0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52"
Jul 20 10:35:38.425393 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.425361    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52"} err="failed to get container status \"0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52\": rpc error: code = NotFound desc = could not find container \"0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52\": container with ID starting with 0a314d215ae9496578840ad5f5eea4482496ae300f50fb81a6648167461e1c52 not found: ID does not exist"
Jul 20 10:35:38.425393 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.425376    2561 scope.go:117] "RemoveContainer" containerID="789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1"
Jul 20 10:35:38.425622 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.425603    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-f69bfc6b4-tmz6b"]
Jul 20 10:35:38.425680 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:35:38.425625    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1\": container with ID starting with 789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1 not found: ID does not exist" containerID="789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1"
Jul 20 10:35:38.425680 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.425648    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1"} err="failed to get container status \"789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1\": rpc error: code = NotFound desc = could not find container \"789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1\": container with ID starting with 789813cfe71a244baa338f68fa92b964c48e8f9dd3e4724646a978f120711da1 not found: ID does not exist"
Jul 20 10:35:38.425680 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.425664    2561 scope.go:117] "RemoveContainer" containerID="96f25bf86658769edc0af59f660f1e8f6ba88b71e8df36636e14c99c75924eb6"
Jul 20 10:35:38.425903 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:35:38.425888    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f25bf86658769edc0af59f660f1e8f6ba88b71e8df36636e14c99c75924eb6\": container with ID starting with 96f25bf86658769edc0af59f660f1e8f6ba88b71e8df36636e14c99c75924eb6 not found: ID does not exist" containerID="96f25bf86658769edc0af59f660f1e8f6ba88b71e8df36636e14c99c75924eb6"
Jul 20 10:35:38.425958 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.425905    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f25bf86658769edc0af59f660f1e8f6ba88b71e8df36636e14c99c75924eb6"} err="failed to get container status \"96f25bf86658769edc0af59f660f1e8f6ba88b71e8df36636e14c99c75924eb6\": rpc error: code = NotFound desc = could not find container \"96f25bf86658769edc0af59f660f1e8f6ba88b71e8df36636e14c99c75924eb6\": container with ID starting with 96f25bf86658769edc0af59f660f1e8f6ba88b71e8df36636e14c99c75924eb6 not found: ID does not exist"
Jul 20 10:35:38.766637 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:38.766555    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" path="/var/lib/kubelet/pods/d0c9dd00-01dd-402b-a437-05e70e7a82a8/volumes"
Jul 20 10:35:47.379273 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:47.379234    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 20 10:35:57.378827 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:35:57.378744    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 20 10:36:07.379435 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:07.379394    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 20 10:36:17.379268 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:17.379232    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:36:24.532184 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:24.532148    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"]
Jul 20 10:36:24.532615 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:24.532375    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kserve-container" containerID="cri-o://40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d" gracePeriod=30
Jul 20 10:36:24.532615 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:24.532425    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kube-rbac-proxy" containerID="cri-o://7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a" gracePeriod=30
Jul 20 10:36:25.570384 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:25.570353    2561 generic.go:356] "Generic (PLEG): container finished" podID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerID="7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a" exitCode=2
Jul 20 10:36:25.570770 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:25.570432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" event={"ID":"476ed2a3-72a1-4399-9322-4fd213a5c8bc","Type":"ContainerDied","Data":"7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a"}
Jul 20 10:36:27.078449 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.078425    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:36:27.161655 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.161629    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kube-rbac-proxy-sar-config\") pod \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") "
Jul 20 10:36:27.161831 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.161712    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/476ed2a3-72a1-4399-9322-4fd213a5c8bc-cabundle-cert\") pod \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") "
Jul 20 10:36:27.161831 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.161740    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kserve-provision-location\") pod \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") "
Jul 20 10:36:27.161831 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.161765    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/476ed2a3-72a1-4399-9322-4fd213a5c8bc-proxy-tls\") pod \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") "
Jul 20 10:36:27.161831 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.161803    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf6jn\" (UniqueName: \"kubernetes.io/projected/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kube-api-access-hf6jn\") pod \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\" (UID: \"476ed2a3-72a1-4399-9322-4fd213a5c8bc\") "
Jul 20 10:36:27.162046 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.162021    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "476ed2a3-72a1-4399-9322-4fd213a5c8bc" (UID: "476ed2a3-72a1-4399-9322-4fd213a5c8bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:36:27.162101 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.162076    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476ed2a3-72a1-4399-9322-4fd213a5c8bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "476ed2a3-72a1-4399-9322-4fd213a5c8bc" (UID: "476ed2a3-72a1-4399-9322-4fd213a5c8bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:36:27.163799 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.163771    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kube-api-access-hf6jn" (OuterVolumeSpecName: "kube-api-access-hf6jn") pod "476ed2a3-72a1-4399-9322-4fd213a5c8bc" (UID: "476ed2a3-72a1-4399-9322-4fd213a5c8bc"). InnerVolumeSpecName "kube-api-access-hf6jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:36:27.163897 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.163818    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/476ed2a3-72a1-4399-9322-4fd213a5c8bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "476ed2a3-72a1-4399-9322-4fd213a5c8bc" (UID: "476ed2a3-72a1-4399-9322-4fd213a5c8bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:36:27.170772 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.170751    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "476ed2a3-72a1-4399-9322-4fd213a5c8bc" (UID: "476ed2a3-72a1-4399-9322-4fd213a5c8bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:36:27.262315 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.262274    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/476ed2a3-72a1-4399-9322-4fd213a5c8bc-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:36:27.262315 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.262314    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:36:27.262315 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.262325    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/476ed2a3-72a1-4399-9322-4fd213a5c8bc-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:36:27.262522 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.262335    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf6jn\" (UniqueName: \"kubernetes.io/projected/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kube-api-access-hf6jn\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:36:27.262522 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.262345    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/476ed2a3-72a1-4399-9322-4fd213a5c8bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:36:27.583083 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.582992    2561 generic.go:356] "Generic (PLEG): container finished" podID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerID="40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d" exitCode=0
Jul 20 10:36:27.583083 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.583046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" event={"ID":"476ed2a3-72a1-4399-9322-4fd213a5c8bc","Type":"ContainerDied","Data":"40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d"}
Jul 20 10:36:27.583083 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.583058    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"
Jul 20 10:36:27.583372 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.583135    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82" event={"ID":"476ed2a3-72a1-4399-9322-4fd213a5c8bc","Type":"ContainerDied","Data":"6fcff64e5e8199f1d19a0c10059f7a111b9f4905b8c1782adb953f1c41cd7ab5"}
Jul 20 10:36:27.583372 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.583162    2561 scope.go:117] "RemoveContainer" containerID="7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a"
Jul 20 10:36:27.591233 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.591213    2561 scope.go:117] "RemoveContainer" containerID="40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d"
Jul 20 10:36:27.598045 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.598029    2561 scope.go:117] "RemoveContainer" containerID="172ec67fcc383319a09e47cc61c4ab6d1f84003b6792412e80c0f00efdc74502"
Jul 20 10:36:27.605504 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.605476    2561 scope.go:117] "RemoveContainer" containerID="7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a"
Jul 20 10:36:27.605767 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:36:27.605749    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a\": container with ID starting with 7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a not found: ID does not exist" containerID="7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a"
Jul 20 10:36:27.605817 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.605776    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a"} err="failed to get container status \"7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a\": rpc error: code = NotFound desc = could not find container \"7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a\": container with ID starting with 7b41404d17c21d649d35b1d9970e60c538b4bec9d2e8905465cc2753087c6d9a not found: ID does not exist"
Jul 20 10:36:27.605817 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.605793    2561 scope.go:117] "RemoveContainer" containerID="40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d"
Jul 20 10:36:27.606077 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:36:27.606057    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d\": container with ID starting with 40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d not found: ID does not exist" containerID="40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d"
Jul 20 10:36:27.606164 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.606086    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d"} err="failed to get container status \"40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d\": rpc error: code = NotFound desc = could not find container \"40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d\": container with ID starting with 40917ff0e4938c7db917b91517b200753c395546814c3dd24e7bfa89f394ca8d not found: ID does not exist"
Jul 20 10:36:27.606164 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.606107    2561 scope.go:117] "RemoveContainer" containerID="172ec67fcc383319a09e47cc61c4ab6d1f84003b6792412e80c0f00efdc74502"
Jul 20 10:36:27.606402 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:36:27.606383    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172ec67fcc383319a09e47cc61c4ab6d1f84003b6792412e80c0f00efdc74502\": container with ID starting with 172ec67fcc383319a09e47cc61c4ab6d1f84003b6792412e80c0f00efdc74502 not found: ID does not exist" containerID="172ec67fcc383319a09e47cc61c4ab6d1f84003b6792412e80c0f00efdc74502"
Jul 20 10:36:27.606455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.606407    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172ec67fcc383319a09e47cc61c4ab6d1f84003b6792412e80c0f00efdc74502"} err="failed to get container status \"172ec67fcc383319a09e47cc61c4ab6d1f84003b6792412e80c0f00efdc74502\": rpc error: code = NotFound desc = could not find container \"172ec67fcc383319a09e47cc61c4ab6d1f84003b6792412e80c0f00efdc74502\": container with ID starting with 172ec67fcc383319a09e47cc61c4ab6d1f84003b6792412e80c0f00efdc74502 not found: ID does not exist"
Jul 20 10:36:27.606675 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.606661    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"]
Jul 20 10:36:27.610384 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:27.610365    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b6h82"]
Jul 20 10:36:28.765003 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:28.764971    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" path="/var/lib/kubelet/pods/476ed2a3-72a1-4399-9322-4fd213a5c8bc/volumes"
Jul 20 10:36:29.674530 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674497    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"]
Jul 20 10:36:29.674759 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674746    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kserve-agent"
Jul 20 10:36:29.674801 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674761    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kserve-agent"
Jul 20 10:36:29.674801 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674770    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="storage-initializer"
Jul 20 10:36:29.674801 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674777    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="storage-initializer"
Jul 20 10:36:29.674801 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674784    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kserve-container"
Jul 20 10:36:29.674801 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674790    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kserve-container"
Jul 20 10:36:29.674801 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674800    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kube-rbac-proxy"
Jul 20 10:36:29.674983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674805    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kube-rbac-proxy"
Jul 20 10:36:29.674983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674815    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="storage-initializer"
Jul 20 10:36:29.674983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674820    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="storage-initializer"
Jul 20 10:36:29.674983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674825    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kserve-container"
Jul 20 10:36:29.674983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674830    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kserve-container"
Jul 20 10:36:29.674983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674837    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kube-rbac-proxy"
Jul 20 10:36:29.674983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674842    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kube-rbac-proxy"
Jul 20 10:36:29.674983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674878    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kube-rbac-proxy"
Jul 20 10:36:29.674983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674885    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kserve-container"
Jul 20 10:36:29.674983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674891    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kube-rbac-proxy"
Jul 20 10:36:29.674983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674898    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c9dd00-01dd-402b-a437-05e70e7a82a8" containerName="kserve-agent"
Jul 20 10:36:29.674983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.674904    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="476ed2a3-72a1-4399-9322-4fd213a5c8bc" containerName="kserve-container"
Jul 20 10:36:29.678000 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.677981    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.680561 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.680540    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 20 10:36:29.680698 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.680680    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 20 10:36:29.681069 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.681028    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:36:29.681069 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.681045    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:36:29.681287 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.681069    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:36:29.681731 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.681712    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:36:29.681810 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.681747    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:36:29.690951 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.690931    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"]
Jul 20 10:36:29.782173 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.782143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4lwq\" (UniqueName: \"kubernetes.io/projected/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kube-api-access-w4lwq\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.782563 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.782184    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c271b61e-c98a-48fb-a14f-43766e1fdaaf-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.782563 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.782214    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.782563 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.782288    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.782563 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.782324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c271b61e-c98a-48fb-a14f-43766e1fdaaf-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.883518 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.883489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4lwq\" (UniqueName: \"kubernetes.io/projected/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kube-api-access-w4lwq\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.883660 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.883522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c271b61e-c98a-48fb-a14f-43766e1fdaaf-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.883660 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.883540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.883660 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.883568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.883660 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.883593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c271b61e-c98a-48fb-a14f-43766e1fdaaf-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.883880 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:36:29.883718    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 20 10:36:29.883880 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:36:29.883782    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c271b61e-c98a-48fb-a14f-43766e1fdaaf-proxy-tls podName:c271b61e-c98a-48fb-a14f-43766e1fdaaf nodeName:}" failed. No retries permitted until 2026-07-20 10:36:30.383762546 +0000 UTC m=+2370.221861113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c271b61e-c98a-48fb-a14f-43766e1fdaaf-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-qzhnn" (UID: "c271b61e-c98a-48fb-a14f-43766e1fdaaf") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 20 10:36:29.884037 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.884019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.884214 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.884195    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c271b61e-c98a-48fb-a14f-43766e1fdaaf-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.884323 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.884306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:29.892467 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:29.892444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4lwq\" (UniqueName: \"kubernetes.io/projected/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kube-api-access-w4lwq\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:30.386940 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:30.386898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c271b61e-c98a-48fb-a14f-43766e1fdaaf-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:30.389287 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:30.389268    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c271b61e-c98a-48fb-a14f-43766e1fdaaf-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qzhnn\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:30.588107 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:30.588066    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:30.717569 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:30.717546    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"]
Jul 20 10:36:30.719308 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:36:30.719279    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc271b61e_c98a_48fb_a14f_43766e1fdaaf.slice/crio-50456158ada8c83167c9bf64901d22587d58600453d7049eb2261183ef86cadf WatchSource:0}: Error finding container 50456158ada8c83167c9bf64901d22587d58600453d7049eb2261183ef86cadf: Status 404 returned error can't find the container with id 50456158ada8c83167c9bf64901d22587d58600453d7049eb2261183ef86cadf
Jul 20 10:36:31.600132 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:31.600083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" event={"ID":"c271b61e-c98a-48fb-a14f-43766e1fdaaf","Type":"ContainerStarted","Data":"ab7758727af0cd3f2b169e2dcdd0074198c88970d2a0c9bf6e024dce85e8a127"}
Jul 20 10:36:31.600504 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:31.600157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" event={"ID":"c271b61e-c98a-48fb-a14f-43766e1fdaaf","Type":"ContainerStarted","Data":"50456158ada8c83167c9bf64901d22587d58600453d7049eb2261183ef86cadf"}
Jul 20 10:36:35.615031 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:35.614995    2561 generic.go:356] "Generic (PLEG): container finished" podID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerID="ab7758727af0cd3f2b169e2dcdd0074198c88970d2a0c9bf6e024dce85e8a127" exitCode=0
Jul 20 10:36:35.615477 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:35.615029    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" event={"ID":"c271b61e-c98a-48fb-a14f-43766e1fdaaf","Type":"ContainerDied","Data":"ab7758727af0cd3f2b169e2dcdd0074198c88970d2a0c9bf6e024dce85e8a127"}
Jul 20 10:36:36.621259 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:36.621226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" event={"ID":"c271b61e-c98a-48fb-a14f-43766e1fdaaf","Type":"ContainerStarted","Data":"0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e"}
Jul 20 10:36:36.621645 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:36.621284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" event={"ID":"c271b61e-c98a-48fb-a14f-43766e1fdaaf","Type":"ContainerStarted","Data":"84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266"}
Jul 20 10:36:36.621645 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:36.621546    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:36.621719 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:36.621684    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:36.622924 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:36.622897    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 20 10:36:36.644015 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:36.643976    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" podStartSLOduration=3.49643909 podStartE2EDuration="7.643963766s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.147524676s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:36:36.642251655 +0000 UTC m=+2376.480350260" watchObservedRunningTime="2026-07-20 10:36:36.643963766 +0000 UTC m=+2376.482062353"
Jul 20 10:36:37.624563 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:37.624519    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 20 10:36:42.628766 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:42.628732    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:36:42.629351 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:42.629322    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 20 10:36:52.629727 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:36:52.629685    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 20 10:37:02.629457 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:02.629420    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 20 10:37:12.630045 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:12.630009    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 20 10:37:16.098340 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:16.098302    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:37:16.101606 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:16.101588    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:37:22.629763 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:22.629733    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:37:31.341760 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:31.341731    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"]
Jul 20 10:37:31.342174 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:31.341974    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kserve-container" containerID="cri-o://84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266" gracePeriod=30
Jul 20 10:37:31.342174 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:31.342000    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kube-rbac-proxy" containerID="cri-o://0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e" gracePeriod=30
Jul 20 10:37:31.823548 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:31.823515    2561 generic.go:356] "Generic (PLEG): container finished" podID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerID="0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e" exitCode=2
Jul 20 10:37:31.823704 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:31.823571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" event={"ID":"c271b61e-c98a-48fb-a14f-43766e1fdaaf","Type":"ContainerDied","Data":"0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e"}
Jul 20 10:37:32.625590 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:32.625548    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Jul 20 10:37:32.629949 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:32.629922    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 20 10:37:33.789004 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.788978    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:37:33.832433 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.832367    2561 generic.go:356] "Generic (PLEG): container finished" podID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerID="84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266" exitCode=0
Jul 20 10:37:33.832433 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.832416    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"
Jul 20 10:37:33.832715 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.832419    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" event={"ID":"c271b61e-c98a-48fb-a14f-43766e1fdaaf","Type":"ContainerDied","Data":"84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266"}
Jul 20 10:37:33.832715 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.832544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn" event={"ID":"c271b61e-c98a-48fb-a14f-43766e1fdaaf","Type":"ContainerDied","Data":"50456158ada8c83167c9bf64901d22587d58600453d7049eb2261183ef86cadf"}
Jul 20 10:37:33.832715 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.832574    2561 scope.go:117] "RemoveContainer" containerID="0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e"
Jul 20 10:37:33.839864 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.839845    2561 scope.go:117] "RemoveContainer" containerID="84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266"
Jul 20 10:37:33.846786 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.846770    2561 scope.go:117] "RemoveContainer" containerID="ab7758727af0cd3f2b169e2dcdd0074198c88970d2a0c9bf6e024dce85e8a127"
Jul 20 10:37:33.853358 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.853340    2561 scope.go:117] "RemoveContainer" containerID="0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e"
Jul 20 10:37:33.853609 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:37:33.853592    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e\": container with ID starting with 0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e not found: ID does not exist" containerID="0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e"
Jul 20 10:37:33.853662 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.853621    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e"} err="failed to get container status \"0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e\": rpc error: code = NotFound desc = could not find container \"0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e\": container with ID starting with 0a72a403b241fe322cbf047f09439a89a1bbb789cfb6af8d7aa562419e1cf54e not found: ID does not exist"
Jul 20 10:37:33.853662 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.853638    2561 scope.go:117] "RemoveContainer" containerID="84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266"
Jul 20 10:37:33.853863 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:37:33.853848    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266\": container with ID starting with 84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266 not found: ID does not exist" containerID="84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266"
Jul 20 10:37:33.853903 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.853867    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266"} err="failed to get container status \"84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266\": rpc error: code = NotFound desc = could not find container \"84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266\": container with ID starting with 84703c0891c971e0495bfca2cf83ae67305262b263f1cbbab33585f975333266 not found: ID does not exist"
Jul 20 10:37:33.853903 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.853889    2561 scope.go:117] "RemoveContainer" containerID="ab7758727af0cd3f2b169e2dcdd0074198c88970d2a0c9bf6e024dce85e8a127"
Jul 20 10:37:33.854107 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:37:33.854091    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7758727af0cd3f2b169e2dcdd0074198c88970d2a0c9bf6e024dce85e8a127\": container with ID starting with ab7758727af0cd3f2b169e2dcdd0074198c88970d2a0c9bf6e024dce85e8a127 not found: ID does not exist" containerID="ab7758727af0cd3f2b169e2dcdd0074198c88970d2a0c9bf6e024dce85e8a127"
Jul 20 10:37:33.854168 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.854131    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7758727af0cd3f2b169e2dcdd0074198c88970d2a0c9bf6e024dce85e8a127"} err="failed to get container status \"ab7758727af0cd3f2b169e2dcdd0074198c88970d2a0c9bf6e024dce85e8a127\": rpc error: code = NotFound desc = could not find container \"ab7758727af0cd3f2b169e2dcdd0074198c88970d2a0c9bf6e024dce85e8a127\": container with ID starting with ab7758727af0cd3f2b169e2dcdd0074198c88970d2a0c9bf6e024dce85e8a127 not found: ID does not exist"
Jul 20 10:37:33.938414 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.938388    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kube-rbac-proxy-sar-config\") pod \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") "
Jul 20 10:37:33.938523 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.938441    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4lwq\" (UniqueName: \"kubernetes.io/projected/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kube-api-access-w4lwq\") pod \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") "
Jul 20 10:37:33.938523 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.938477    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c271b61e-c98a-48fb-a14f-43766e1fdaaf-proxy-tls\") pod \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") "
Jul 20 10:37:33.938622 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.938528    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kserve-provision-location\") pod \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") "
Jul 20 10:37:33.938622 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.938559    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c271b61e-c98a-48fb-a14f-43766e1fdaaf-cabundle-cert\") pod \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\" (UID: \"c271b61e-c98a-48fb-a14f-43766e1fdaaf\") "
Jul 20 10:37:33.938860 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.938829    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c271b61e-c98a-48fb-a14f-43766e1fdaaf" (UID: "c271b61e-c98a-48fb-a14f-43766e1fdaaf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:37:33.938959 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.938906    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c271b61e-c98a-48fb-a14f-43766e1fdaaf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c271b61e-c98a-48fb-a14f-43766e1fdaaf" (UID: "c271b61e-c98a-48fb-a14f-43766e1fdaaf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:37:33.940396 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.940375    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c271b61e-c98a-48fb-a14f-43766e1fdaaf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c271b61e-c98a-48fb-a14f-43766e1fdaaf" (UID: "c271b61e-c98a-48fb-a14f-43766e1fdaaf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:37:33.940503 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.940484    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kube-api-access-w4lwq" (OuterVolumeSpecName: "kube-api-access-w4lwq") pod "c271b61e-c98a-48fb-a14f-43766e1fdaaf" (UID: "c271b61e-c98a-48fb-a14f-43766e1fdaaf"). InnerVolumeSpecName "kube-api-access-w4lwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:37:33.948331 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:33.948305    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c271b61e-c98a-48fb-a14f-43766e1fdaaf" (UID: "c271b61e-c98a-48fb-a14f-43766e1fdaaf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:37:34.039782 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:34.039755    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c271b61e-c98a-48fb-a14f-43766e1fdaaf-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:37:34.039782 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:34.039780    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:37:34.039935 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:34.039793    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c271b61e-c98a-48fb-a14f-43766e1fdaaf-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:37:34.039935 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:34.039805    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:37:34.039935 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:34.039817    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4lwq\" (UniqueName: \"kubernetes.io/projected/c271b61e-c98a-48fb-a14f-43766e1fdaaf-kube-api-access-w4lwq\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:37:34.164633 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:34.164604    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"]
Jul 20 10:37:34.167533 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:34.167510    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qzhnn"]
Jul 20 10:37:34.764793 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:34.764757    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" path="/var/lib/kubelet/pods/c271b61e-c98a-48fb-a14f-43766e1fdaaf/volumes"
Jul 20 10:37:36.513019 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.512986    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"]
Jul 20 10:37:36.513492 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.513361    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="storage-initializer"
Jul 20 10:37:36.513492 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.513379    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="storage-initializer"
Jul 20 10:37:36.513492 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.513407    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kube-rbac-proxy"
Jul 20 10:37:36.513492 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.513416    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kube-rbac-proxy"
Jul 20 10:37:36.513492 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.513431    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kserve-container"
Jul 20 10:37:36.513492 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.513439    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kserve-container"
Jul 20 10:37:36.513825 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.513500    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kserve-container"
Jul 20 10:37:36.513825 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.513514    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c271b61e-c98a-48fb-a14f-43766e1fdaaf" containerName="kube-rbac-proxy"
Jul 20 10:37:36.518156 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.518133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.521590 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.521560    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:37:36.521758 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.521680    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 20 10:37:36.521839 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.521769    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:37:36.521932 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.521893    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 20 10:37:36.521999 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.521984    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:37:36.523219 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.523202    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:37:36.523319 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.523256    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:37:36.527756 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.527733    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"]
Jul 20 10:37:36.661617 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.661587    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prgx7\" (UniqueName: \"kubernetes.io/projected/724784d6-79f6-4a27-b5c7-b801f4ad825a-kube-api-access-prgx7\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.661768 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.661624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/724784d6-79f6-4a27-b5c7-b801f4ad825a-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.661768 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.661642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/724784d6-79f6-4a27-b5c7-b801f4ad825a-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.661768 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.661743    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/724784d6-79f6-4a27-b5c7-b801f4ad825a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.661897 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.661769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/724784d6-79f6-4a27-b5c7-b801f4ad825a-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.762088 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.762058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/724784d6-79f6-4a27-b5c7-b801f4ad825a-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.762088 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.762092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/724784d6-79f6-4a27-b5c7-b801f4ad825a-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.762377 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.762157    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/724784d6-79f6-4a27-b5c7-b801f4ad825a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.762377 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.762173    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/724784d6-79f6-4a27-b5c7-b801f4ad825a-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.762377 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.762224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prgx7\" (UniqueName: \"kubernetes.io/projected/724784d6-79f6-4a27-b5c7-b801f4ad825a-kube-api-access-prgx7\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.762840 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.762814    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/724784d6-79f6-4a27-b5c7-b801f4ad825a-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.762966 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.762907    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/724784d6-79f6-4a27-b5c7-b801f4ad825a-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.762966 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.762933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/724784d6-79f6-4a27-b5c7-b801f4ad825a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.764912 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.764855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/724784d6-79f6-4a27-b5c7-b801f4ad825a-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.774422 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.774398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prgx7\" (UniqueName: \"kubernetes.io/projected/724784d6-79f6-4a27-b5c7-b801f4ad825a-kube-api-access-prgx7\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.828055 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.828031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:36.955651 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:36.955624    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"]
Jul 20 10:37:36.956816 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:37:36.956791    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724784d6_79f6_4a27_b5c7_b801f4ad825a.slice/crio-839d093e9c2b7720af2d55cf2a2bcc67904be11f1528ecadfb6a781bf3f6fba7 WatchSource:0}: Error finding container 839d093e9c2b7720af2d55cf2a2bcc67904be11f1528ecadfb6a781bf3f6fba7: Status 404 returned error can't find the container with id 839d093e9c2b7720af2d55cf2a2bcc67904be11f1528ecadfb6a781bf3f6fba7
Jul 20 10:37:37.849232 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:37.849198    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" event={"ID":"724784d6-79f6-4a27-b5c7-b801f4ad825a","Type":"ContainerStarted","Data":"3a51a2bc0ba5707ca8bfbf467849c45e2c53402ba84a9c42431542216f61201d"}
Jul 20 10:37:37.849603 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:37.849255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" event={"ID":"724784d6-79f6-4a27-b5c7-b801f4ad825a","Type":"ContainerStarted","Data":"839d093e9c2b7720af2d55cf2a2bcc67904be11f1528ecadfb6a781bf3f6fba7"}
Jul 20 10:37:41.863832 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:41.863803    2561 generic.go:356] "Generic (PLEG): container finished" podID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerID="3a51a2bc0ba5707ca8bfbf467849c45e2c53402ba84a9c42431542216f61201d" exitCode=0
Jul 20 10:37:41.864216 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:41.863877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" event={"ID":"724784d6-79f6-4a27-b5c7-b801f4ad825a","Type":"ContainerDied","Data":"3a51a2bc0ba5707ca8bfbf467849c45e2c53402ba84a9c42431542216f61201d"}
Jul 20 10:37:42.870977 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:42.870935    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" event={"ID":"724784d6-79f6-4a27-b5c7-b801f4ad825a","Type":"ContainerStarted","Data":"610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7"}
Jul 20 10:37:42.871396 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:42.871015    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" event={"ID":"724784d6-79f6-4a27-b5c7-b801f4ad825a","Type":"ContainerStarted","Data":"b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac"}
Jul 20 10:37:42.871396 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:42.871261    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:42.894463 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:42.894415    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" podStartSLOduration=2.513367379 podStartE2EDuration="6.894401054s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.381033675s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:37:42.892284563 +0000 UTC m=+2442.730383168" watchObservedRunningTime="2026-07-20 10:37:42.894401054 +0000 UTC m=+2442.732499638"
Jul 20 10:37:43.875099 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:43.875070    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:43.876270 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:43.876240    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 20 10:37:44.878087 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:44.878045    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 20 10:37:49.882661 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:49.882632    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:37:49.883245 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:49.883212    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 20 10:37:59.883092 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:37:59.883051    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 20 10:38:09.883421 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:09.883379    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 20 10:38:19.883454 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:19.883414    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 20 10:38:29.883298 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:29.883264    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:38:38.516672 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:38.516638    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"]
Jul 20 10:38:38.517174 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:38.516902    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kserve-container" containerID="cri-o://b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac" gracePeriod=30
Jul 20 10:38:38.517174 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:38.516910    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kube-rbac-proxy" containerID="cri-o://610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7" gracePeriod=30
Jul 20 10:38:39.076043 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:39.076002    2561 generic.go:356] "Generic (PLEG): container finished" podID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerID="610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7" exitCode=2
Jul 20 10:38:39.076222 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:39.076070    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" event={"ID":"724784d6-79f6-4a27-b5c7-b801f4ad825a","Type":"ContainerDied","Data":"610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7"}
Jul 20 10:38:39.879276 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:39.879233    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Jul 20 10:38:39.883591 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:39.883565    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 20 10:38:41.064361 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.064339    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:38:41.085231 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.085200    2561 generic.go:356] "Generic (PLEG): container finished" podID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerID="b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac" exitCode=0
Jul 20 10:38:41.085341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.085256    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"
Jul 20 10:38:41.085341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.085283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" event={"ID":"724784d6-79f6-4a27-b5c7-b801f4ad825a","Type":"ContainerDied","Data":"b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac"}
Jul 20 10:38:41.085414 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.085341    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg" event={"ID":"724784d6-79f6-4a27-b5c7-b801f4ad825a","Type":"ContainerDied","Data":"839d093e9c2b7720af2d55cf2a2bcc67904be11f1528ecadfb6a781bf3f6fba7"}
Jul 20 10:38:41.085414 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.085368    2561 scope.go:117] "RemoveContainer" containerID="610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7"
Jul 20 10:38:41.094040 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.094021    2561 scope.go:117] "RemoveContainer" containerID="b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac"
Jul 20 10:38:41.098189 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.098166    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/724784d6-79f6-4a27-b5c7-b801f4ad825a-kube-rbac-proxy-sar-config\") pod \"724784d6-79f6-4a27-b5c7-b801f4ad825a\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") "
Jul 20 10:38:41.098300 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.098216    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/724784d6-79f6-4a27-b5c7-b801f4ad825a-kserve-provision-location\") pod \"724784d6-79f6-4a27-b5c7-b801f4ad825a\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") "
Jul 20 10:38:41.098300 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.098284    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/724784d6-79f6-4a27-b5c7-b801f4ad825a-proxy-tls\") pod \"724784d6-79f6-4a27-b5c7-b801f4ad825a\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") "
Jul 20 10:38:41.098394 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.098324    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prgx7\" (UniqueName: \"kubernetes.io/projected/724784d6-79f6-4a27-b5c7-b801f4ad825a-kube-api-access-prgx7\") pod \"724784d6-79f6-4a27-b5c7-b801f4ad825a\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") "
Jul 20 10:38:41.098394 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.098355    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/724784d6-79f6-4a27-b5c7-b801f4ad825a-cabundle-cert\") pod \"724784d6-79f6-4a27-b5c7-b801f4ad825a\" (UID: \"724784d6-79f6-4a27-b5c7-b801f4ad825a\") "
Jul 20 10:38:41.098657 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.098630    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724784d6-79f6-4a27-b5c7-b801f4ad825a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "724784d6-79f6-4a27-b5c7-b801f4ad825a" (UID: "724784d6-79f6-4a27-b5c7-b801f4ad825a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:38:41.098788 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.098766    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724784d6-79f6-4a27-b5c7-b801f4ad825a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "724784d6-79f6-4a27-b5c7-b801f4ad825a" (UID: "724784d6-79f6-4a27-b5c7-b801f4ad825a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:38:41.100907 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.100803    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724784d6-79f6-4a27-b5c7-b801f4ad825a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "724784d6-79f6-4a27-b5c7-b801f4ad825a" (UID: "724784d6-79f6-4a27-b5c7-b801f4ad825a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:38:41.100907 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.100889    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724784d6-79f6-4a27-b5c7-b801f4ad825a-kube-api-access-prgx7" (OuterVolumeSpecName: "kube-api-access-prgx7") pod "724784d6-79f6-4a27-b5c7-b801f4ad825a" (UID: "724784d6-79f6-4a27-b5c7-b801f4ad825a"). InnerVolumeSpecName "kube-api-access-prgx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:38:41.103320 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.103305    2561 scope.go:117] "RemoveContainer" containerID="3a51a2bc0ba5707ca8bfbf467849c45e2c53402ba84a9c42431542216f61201d"
Jul 20 10:38:41.109035 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.109012    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724784d6-79f6-4a27-b5c7-b801f4ad825a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "724784d6-79f6-4a27-b5c7-b801f4ad825a" (UID: "724784d6-79f6-4a27-b5c7-b801f4ad825a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:38:41.113530 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.113505    2561 scope.go:117] "RemoveContainer" containerID="610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7"
Jul 20 10:38:41.113788 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:38:41.113766    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7\": container with ID starting with 610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7 not found: ID does not exist" containerID="610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7"
Jul 20 10:38:41.113846 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.113798    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7"} err="failed to get container status \"610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7\": rpc error: code = NotFound desc = could not find container \"610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7\": container with ID starting with 610e0f817fbe92ebe4673eed1d3eef7ff64dfcff4175f708de8ffb32ab7b48e7 not found: ID does not exist"
Jul 20 10:38:41.113846 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.113816    2561 scope.go:117] "RemoveContainer" containerID="b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac"
Jul 20 10:38:41.114096 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:38:41.114081    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac\": container with ID starting with b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac not found: ID does not exist" containerID="b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac"
Jul 20 10:38:41.114179 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.114099    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac"} err="failed to get container status \"b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac\": rpc error: code = NotFound desc = could not find container \"b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac\": container with ID starting with b0701278312dbacf5fcd2bc7d1c2e110fd030f1a828291018707a6bef8593fac not found: ID does not exist"
Jul 20 10:38:41.114179 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.114134    2561 scope.go:117] "RemoveContainer" containerID="3a51a2bc0ba5707ca8bfbf467849c45e2c53402ba84a9c42431542216f61201d"
Jul 20 10:38:41.114353 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:38:41.114337    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a51a2bc0ba5707ca8bfbf467849c45e2c53402ba84a9c42431542216f61201d\": container with ID starting with 3a51a2bc0ba5707ca8bfbf467849c45e2c53402ba84a9c42431542216f61201d not found: ID does not exist" containerID="3a51a2bc0ba5707ca8bfbf467849c45e2c53402ba84a9c42431542216f61201d"
Jul 20 10:38:41.114390 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.114367    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a51a2bc0ba5707ca8bfbf467849c45e2c53402ba84a9c42431542216f61201d"} err="failed to get container status \"3a51a2bc0ba5707ca8bfbf467849c45e2c53402ba84a9c42431542216f61201d\": rpc error: code = NotFound desc = could not find container \"3a51a2bc0ba5707ca8bfbf467849c45e2c53402ba84a9c42431542216f61201d\": container with ID starting with 3a51a2bc0ba5707ca8bfbf467849c45e2c53402ba84a9c42431542216f61201d not found: ID does not exist"
Jul 20 10:38:41.199709 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.199659    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/724784d6-79f6-4a27-b5c7-b801f4ad825a-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:38:41.199709 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.199679    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prgx7\" (UniqueName: \"kubernetes.io/projected/724784d6-79f6-4a27-b5c7-b801f4ad825a-kube-api-access-prgx7\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:38:41.199709 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.199698    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/724784d6-79f6-4a27-b5c7-b801f4ad825a-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:38:41.199709 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.199708    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/724784d6-79f6-4a27-b5c7-b801f4ad825a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:38:41.199890 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.199717    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/724784d6-79f6-4a27-b5c7-b801f4ad825a-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:38:41.407643 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.407613    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"]
Jul 20 10:38:41.411656 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:41.411634    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-z8vbg"]
Jul 20 10:38:42.769634 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:42.769597    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" path="/var/lib/kubelet/pods/724784d6-79f6-4a27-b5c7-b801f4ad825a/volumes"
Jul 20 10:38:43.686313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.686276    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"]
Jul 20 10:38:43.686550 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.686537    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kube-rbac-proxy"
Jul 20 10:38:43.686596 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.686552    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kube-rbac-proxy"
Jul 20 10:38:43.686596 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.686563    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kserve-container"
Jul 20 10:38:43.686596 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.686569    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kserve-container"
Jul 20 10:38:43.686596 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.686577    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="storage-initializer"
Jul 20 10:38:43.686596 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.686582    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="storage-initializer"
Jul 20 10:38:43.686749 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.686621    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kserve-container"
Jul 20 10:38:43.686749 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.686629    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="724784d6-79f6-4a27-b5c7-b801f4ad825a" containerName="kube-rbac-proxy"
Jul 20 10:38:43.691500 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.691477    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.694326 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.694293    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:38:43.694326 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.694316    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:38:43.694499 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.694320    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:38:43.694499 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.694320    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 20 10:38:43.695511 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.695492    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 20 10:38:43.695610 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.695523    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:38:43.695610 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.695524    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:38:43.702076 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.702057    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"]
Jul 20 10:38:43.716711 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.716681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79ea2fed-d816-45df-a301-d7a436ec9818-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.716806 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.716762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79ea2fed-d816-45df-a301-d7a436ec9818-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.716806 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.716796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79ea2fed-d816-45df-a301-d7a436ec9818-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.716900 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.716816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79ea2fed-d816-45df-a301-d7a436ec9818-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.716900 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.716857    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjpj\" (UniqueName: \"kubernetes.io/projected/79ea2fed-d816-45df-a301-d7a436ec9818-kube-api-access-lkjpj\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.817141 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.817103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79ea2fed-d816-45df-a301-d7a436ec9818-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.817478 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.817145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79ea2fed-d816-45df-a301-d7a436ec9818-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.817478 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.817163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79ea2fed-d816-45df-a301-d7a436ec9818-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.817478 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.817196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjpj\" (UniqueName: \"kubernetes.io/projected/79ea2fed-d816-45df-a301-d7a436ec9818-kube-api-access-lkjpj\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.817478 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.817213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79ea2fed-d816-45df-a301-d7a436ec9818-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.817652 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.817547    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79ea2fed-d816-45df-a301-d7a436ec9818-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.817802 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.817786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79ea2fed-d816-45df-a301-d7a436ec9818-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.817848 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.817809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79ea2fed-d816-45df-a301-d7a436ec9818-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.819532 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.819516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79ea2fed-d816-45df-a301-d7a436ec9818-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:43.827042 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:43.827021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjpj\" (UniqueName: \"kubernetes.io/projected/79ea2fed-d816-45df-a301-d7a436ec9818-kube-api-access-lkjpj\") pod \"isvc-pmml-predictor-8674795f59-djwmt\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:44.002450 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:44.002391    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:44.130966 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:44.130804    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"]
Jul 20 10:38:44.134841 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:38:44.134805    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ea2fed_d816_45df_a301_d7a436ec9818.slice/crio-df64feac5b114a3ba1b00c8989c2fd101e0e93e9f9c78ae58f6f1281437dbd95 WatchSource:0}: Error finding container df64feac5b114a3ba1b00c8989c2fd101e0e93e9f9c78ae58f6f1281437dbd95: Status 404 returned error can't find the container with id df64feac5b114a3ba1b00c8989c2fd101e0e93e9f9c78ae58f6f1281437dbd95
Jul 20 10:38:45.101771 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:45.101733    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" event={"ID":"79ea2fed-d816-45df-a301-d7a436ec9818","Type":"ContainerStarted","Data":"ed55c3d1ef5cbc8af2d9c14daaa4af8ce277746ca7a7f8a3aa18699858f2db37"}
Jul 20 10:38:45.102270 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:45.101805    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" event={"ID":"79ea2fed-d816-45df-a301-d7a436ec9818","Type":"ContainerStarted","Data":"df64feac5b114a3ba1b00c8989c2fd101e0e93e9f9c78ae58f6f1281437dbd95"}
Jul 20 10:38:48.114515 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:48.114485    2561 generic.go:356] "Generic (PLEG): container finished" podID="79ea2fed-d816-45df-a301-d7a436ec9818" containerID="ed55c3d1ef5cbc8af2d9c14daaa4af8ce277746ca7a7f8a3aa18699858f2db37" exitCode=0
Jul 20 10:38:48.114881 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:48.114532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" event={"ID":"79ea2fed-d816-45df-a301-d7a436ec9818","Type":"ContainerDied","Data":"ed55c3d1ef5cbc8af2d9c14daaa4af8ce277746ca7a7f8a3aa18699858f2db37"}
Jul 20 10:38:55.149381 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:55.149347    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" event={"ID":"79ea2fed-d816-45df-a301-d7a436ec9818","Type":"ContainerStarted","Data":"8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6"}
Jul 20 10:38:55.149839 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:55.149408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" event={"ID":"79ea2fed-d816-45df-a301-d7a436ec9818","Type":"ContainerStarted","Data":"6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9"}
Jul 20 10:38:55.149839 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:55.149597    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:55.178678 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:55.178630    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podStartSLOduration=1.714312337 podStartE2EDuration="12.178616587s" totalImagesPullingTime="6.834733444s" totalInitContainerRuntime="3.629570806s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:38:55.176589001 +0000 UTC m=+2515.014687599" watchObservedRunningTime="2026-07-20 10:38:55.178616587 +0000 UTC m=+2515.016715172"
Jul 20 10:38:56.152974 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:56.152942    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:38:56.154231 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:56.154200    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 20 10:38:57.156821 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:38:57.156779    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 20 10:39:02.161582 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:39:02.161556    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:39:02.162210 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:39:02.162174    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 20 10:39:12.162639 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:39:12.162601    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 20 10:39:22.162350 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:39:22.162311    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 20 10:39:32.162635 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:39:32.162597    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 20 10:39:42.162971 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:39:42.162935    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 20 10:39:52.162050 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:39:52.162015    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 20 10:40:02.162259 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:02.162221    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 20 10:40:12.163220 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:12.163192    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:40:14.858743 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:14.858710    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"]
Jul 20 10:40:14.859105 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:14.858913    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container" containerID="cri-o://6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9" gracePeriod=30
Jul 20 10:40:14.859105 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:14.859002    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kube-rbac-proxy" containerID="cri-o://8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6" gracePeriod=30
Jul 20 10:40:15.448049 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:15.448016    2561 generic.go:356] "Generic (PLEG): container finished" podID="79ea2fed-d816-45df-a301-d7a436ec9818" containerID="8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6" exitCode=2
Jul 20 10:40:15.448235 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:15.448072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" event={"ID":"79ea2fed-d816-45df-a301-d7a436ec9818","Type":"ContainerDied","Data":"8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6"}
Jul 20 10:40:17.157649 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:17.157603    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 20 10:40:18.092971 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.092948    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:40:18.201277 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.201244    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79ea2fed-d816-45df-a301-d7a436ec9818-kube-rbac-proxy-sar-config\") pod \"79ea2fed-d816-45df-a301-d7a436ec9818\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") "
Jul 20 10:40:18.201746 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.201299    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79ea2fed-d816-45df-a301-d7a436ec9818-proxy-tls\") pod \"79ea2fed-d816-45df-a301-d7a436ec9818\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") "
Jul 20 10:40:18.201746 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.201325    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkjpj\" (UniqueName: \"kubernetes.io/projected/79ea2fed-d816-45df-a301-d7a436ec9818-kube-api-access-lkjpj\") pod \"79ea2fed-d816-45df-a301-d7a436ec9818\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") "
Jul 20 10:40:18.201746 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.201345    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79ea2fed-d816-45df-a301-d7a436ec9818-cabundle-cert\") pod \"79ea2fed-d816-45df-a301-d7a436ec9818\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") "
Jul 20 10:40:18.201746 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.201368    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79ea2fed-d816-45df-a301-d7a436ec9818-kserve-provision-location\") pod \"79ea2fed-d816-45df-a301-d7a436ec9818\" (UID: \"79ea2fed-d816-45df-a301-d7a436ec9818\") "
Jul 20 10:40:18.201746 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.201650    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ea2fed-d816-45df-a301-d7a436ec9818-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79ea2fed-d816-45df-a301-d7a436ec9818" (UID: "79ea2fed-d816-45df-a301-d7a436ec9818"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:40:18.201746 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.201698    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ea2fed-d816-45df-a301-d7a436ec9818-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79ea2fed-d816-45df-a301-d7a436ec9818" (UID: "79ea2fed-d816-45df-a301-d7a436ec9818"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:40:18.201989 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.201748    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ea2fed-d816-45df-a301-d7a436ec9818-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79ea2fed-d816-45df-a301-d7a436ec9818" (UID: "79ea2fed-d816-45df-a301-d7a436ec9818"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:40:18.203545 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.203519    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ea2fed-d816-45df-a301-d7a436ec9818-kube-api-access-lkjpj" (OuterVolumeSpecName: "kube-api-access-lkjpj") pod "79ea2fed-d816-45df-a301-d7a436ec9818" (UID: "79ea2fed-d816-45df-a301-d7a436ec9818"). InnerVolumeSpecName "kube-api-access-lkjpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:40:18.203545 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.203522    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ea2fed-d816-45df-a301-d7a436ec9818-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79ea2fed-d816-45df-a301-d7a436ec9818" (UID: "79ea2fed-d816-45df-a301-d7a436ec9818"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:40:18.302743 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.302715    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79ea2fed-d816-45df-a301-d7a436ec9818-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:40:18.302743 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.302739    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkjpj\" (UniqueName: \"kubernetes.io/projected/79ea2fed-d816-45df-a301-d7a436ec9818-kube-api-access-lkjpj\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:40:18.302743 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.302748    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79ea2fed-d816-45df-a301-d7a436ec9818-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:40:18.302951 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.302758    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79ea2fed-d816-45df-a301-d7a436ec9818-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:40:18.302951 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.302767    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79ea2fed-d816-45df-a301-d7a436ec9818-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:40:18.460931 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.460859    2561 generic.go:356] "Generic (PLEG): container finished" podID="79ea2fed-d816-45df-a301-d7a436ec9818" containerID="6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9" exitCode=0
Jul 20 10:40:18.461055 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.460925    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"
Jul 20 10:40:18.461055 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.460942    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" event={"ID":"79ea2fed-d816-45df-a301-d7a436ec9818","Type":"ContainerDied","Data":"6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9"}
Jul 20 10:40:18.461055 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.461012    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt" event={"ID":"79ea2fed-d816-45df-a301-d7a436ec9818","Type":"ContainerDied","Data":"df64feac5b114a3ba1b00c8989c2fd101e0e93e9f9c78ae58f6f1281437dbd95"}
Jul 20 10:40:18.461055 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.461048    2561 scope.go:117] "RemoveContainer" containerID="8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6"
Jul 20 10:40:18.469309 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.469287    2561 scope.go:117] "RemoveContainer" containerID="6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9"
Jul 20 10:40:18.476404 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.476384    2561 scope.go:117] "RemoveContainer" containerID="ed55c3d1ef5cbc8af2d9c14daaa4af8ce277746ca7a7f8a3aa18699858f2db37"
Jul 20 10:40:18.482144 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.482107    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"]
Jul 20 10:40:18.483725 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.483704    2561 scope.go:117] "RemoveContainer" containerID="8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6"
Jul 20 10:40:18.484009 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:40:18.483991    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6\": container with ID starting with 8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6 not found: ID does not exist" containerID="8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6"
Jul 20 10:40:18.484097 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.484022    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6"} err="failed to get container status \"8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6\": rpc error: code = NotFound desc = could not find container \"8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6\": container with ID starting with 8974725c4a1b8331e4c656cc79a7391314f34db56bd69a3d6a9e01778b1c35b6 not found: ID does not exist"
Jul 20 10:40:18.484097 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.484045    2561 scope.go:117] "RemoveContainer" containerID="6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9"
Jul 20 10:40:18.484320 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:40:18.484298    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9\": container with ID starting with 6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9 not found: ID does not exist" containerID="6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9"
Jul 20 10:40:18.484365 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.484327    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9"} err="failed to get container status \"6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9\": rpc error: code = NotFound desc = could not find container \"6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9\": container with ID starting with 6ae7205cd714baa372ef711fd817a607d566487184a8f5cdc388e6c5ec0bb6e9 not found: ID does not exist"
Jul 20 10:40:18.484365 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.484343    2561 scope.go:117] "RemoveContainer" containerID="ed55c3d1ef5cbc8af2d9c14daaa4af8ce277746ca7a7f8a3aa18699858f2db37"
Jul 20 10:40:18.484547 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:40:18.484532    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed55c3d1ef5cbc8af2d9c14daaa4af8ce277746ca7a7f8a3aa18699858f2db37\": container with ID starting with ed55c3d1ef5cbc8af2d9c14daaa4af8ce277746ca7a7f8a3aa18699858f2db37 not found: ID does not exist" containerID="ed55c3d1ef5cbc8af2d9c14daaa4af8ce277746ca7a7f8a3aa18699858f2db37"
Jul 20 10:40:18.484593 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.484549    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed55c3d1ef5cbc8af2d9c14daaa4af8ce277746ca7a7f8a3aa18699858f2db37"} err="failed to get container status \"ed55c3d1ef5cbc8af2d9c14daaa4af8ce277746ca7a7f8a3aa18699858f2db37\": rpc error: code = NotFound desc = could not find container \"ed55c3d1ef5cbc8af2d9c14daaa4af8ce277746ca7a7f8a3aa18699858f2db37\": container with ID starting with ed55c3d1ef5cbc8af2d9c14daaa4af8ce277746ca7a7f8a3aa18699858f2db37 not found: ID does not exist"
Jul 20 10:40:18.488483 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.488459    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-djwmt"]
Jul 20 10:40:18.765768 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:18.765688    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" path="/var/lib/kubelet/pods/79ea2fed-d816-45df-a301-d7a436ec9818/volumes"
Jul 20 10:40:20.046911 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.046881    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"]
Jul 20 10:40:20.047292 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.047201    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container"
Jul 20 10:40:20.047292 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.047219    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container"
Jul 20 10:40:20.047292 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.047236    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="storage-initializer"
Jul 20 10:40:20.047292 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.047245    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="storage-initializer"
Jul 20 10:40:20.047292 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.047268    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kube-rbac-proxy"
Jul 20 10:40:20.047292 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.047279    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kube-rbac-proxy"
Jul 20 10:40:20.047627 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.047348    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kube-rbac-proxy"
Jul 20 10:40:20.047627 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.047361    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ea2fed-d816-45df-a301-d7a436ec9818" containerName="kserve-container"
Jul 20 10:40:20.051634 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.051617    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.054371 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.054344    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:40:20.054371 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.054365    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:40:20.054523 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.054448    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:40:20.054523 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.054439    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:40:20.055514 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.055500    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 20 10:40:20.055592 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.055577    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 20 10:40:20.055657 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.055629    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:40:20.062308 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.062290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"]
Jul 20 10:40:20.117638 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.117604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.117796 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.117650    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03f56ce6-9b4e-4644-b5a0-92fea00a4320-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.117796 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.117675    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5rd7\" (UniqueName: \"kubernetes.io/projected/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kube-api-access-s5rd7\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.117796 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.117693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03f56ce6-9b4e-4644-b5a0-92fea00a4320-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.117796 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.117752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.218134 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.218083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.218302 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.218156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03f56ce6-9b4e-4644-b5a0-92fea00a4320-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.218302 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.218180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5rd7\" (UniqueName: \"kubernetes.io/projected/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kube-api-access-s5rd7\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.218302 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.218197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03f56ce6-9b4e-4644-b5a0-92fea00a4320-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.218302 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.218230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.218691 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.218664    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.218904 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.218886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.218904 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.218897    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03f56ce6-9b4e-4644-b5a0-92fea00a4320-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.220493 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.220470    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03f56ce6-9b4e-4644-b5a0-92fea00a4320-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.231308 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.231285    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5rd7\" (UniqueName: \"kubernetes.io/projected/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kube-api-access-s5rd7\") pod \"isvc-pmml-runtime-predictor-885fc8777-4rc6n\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.362342 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.362307    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:20.492423 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:20.492399    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"]
Jul 20 10:40:20.494911 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:40:20.494882    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f56ce6_9b4e_4644_b5a0_92fea00a4320.slice/crio-4b3f3058744c2b76aaa00e042776733dfe0d4584456634882aec26f03b09443d WatchSource:0}: Error finding container 4b3f3058744c2b76aaa00e042776733dfe0d4584456634882aec26f03b09443d: Status 404 returned error can't find the container with id 4b3f3058744c2b76aaa00e042776733dfe0d4584456634882aec26f03b09443d
Jul 20 10:40:21.474246 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:21.474213    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" event={"ID":"03f56ce6-9b4e-4644-b5a0-92fea00a4320","Type":"ContainerStarted","Data":"fc99740f1b262673d32a2232fedd1d167ae6375872bc78382e710e01a6e1c757"}
Jul 20 10:40:21.474620 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:21.474272    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" event={"ID":"03f56ce6-9b4e-4644-b5a0-92fea00a4320","Type":"ContainerStarted","Data":"4b3f3058744c2b76aaa00e042776733dfe0d4584456634882aec26f03b09443d"}
Jul 20 10:40:24.486361 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:24.486279    2561 generic.go:356] "Generic (PLEG): container finished" podID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerID="fc99740f1b262673d32a2232fedd1d167ae6375872bc78382e710e01a6e1c757" exitCode=0
Jul 20 10:40:24.486720 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:24.486353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" event={"ID":"03f56ce6-9b4e-4644-b5a0-92fea00a4320","Type":"ContainerDied","Data":"fc99740f1b262673d32a2232fedd1d167ae6375872bc78382e710e01a6e1c757"}
Jul 20 10:40:25.491642 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:25.491612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" event={"ID":"03f56ce6-9b4e-4644-b5a0-92fea00a4320","Type":"ContainerStarted","Data":"287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9"}
Jul 20 10:40:25.492010 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:25.491671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" event={"ID":"03f56ce6-9b4e-4644-b5a0-92fea00a4320","Type":"ContainerStarted","Data":"29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1"}
Jul 20 10:40:25.492010 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:25.491846    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:25.514978 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:25.514930    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podStartSLOduration=1.916717652 podStartE2EDuration="5.514914585s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.598196933s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:40:25.514183125 +0000 UTC m=+2605.352281720" watchObservedRunningTime="2026-07-20 10:40:25.514914585 +0000 UTC m=+2605.353013231"
Jul 20 10:40:26.495610 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:26.495578    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:26.496698 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:26.496674    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 20 10:40:27.499866 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:27.499829    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 20 10:40:32.505568 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:32.505542    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:40:32.506006 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:32.505977    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 20 10:40:42.506930 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:42.506890    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 20 10:40:52.506592 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:40:52.506553    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 20 10:41:02.506857 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:02.506817    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 20 10:41:12.506882 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:12.506838    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 20 10:41:22.506680 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:22.506641    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 20 10:41:32.506173 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:32.506128    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 20 10:41:37.761836 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:37.761807    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:41:41.317205 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:41.317172    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"]
Jul 20 10:41:41.317586 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:41.317419    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container" containerID="cri-o://29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1" gracePeriod=30
Jul 20 10:41:41.317586 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:41.317459    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kube-rbac-proxy" containerID="cri-o://287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9" gracePeriod=30
Jul 20 10:41:41.771987 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:41.771954    2561 generic.go:356] "Generic (PLEG): container finished" podID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerID="287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9" exitCode=2
Jul 20 10:41:41.772170 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:41.772030    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" event={"ID":"03f56ce6-9b4e-4644-b5a0-92fea00a4320","Type":"ContainerDied","Data":"287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9"}
Jul 20 10:41:42.500446 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:42.500398    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Jul 20 10:41:44.558678 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.558656    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:41:44.635892 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.635866    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kube-rbac-proxy-sar-config\") pod \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") "
Jul 20 10:41:44.636047 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.635905    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5rd7\" (UniqueName: \"kubernetes.io/projected/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kube-api-access-s5rd7\") pod \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") "
Jul 20 10:41:44.636047 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.635932    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kserve-provision-location\") pod \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") "
Jul 20 10:41:44.636047 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.635983    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03f56ce6-9b4e-4644-b5a0-92fea00a4320-proxy-tls\") pod \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") "
Jul 20 10:41:44.636047 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.636010    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03f56ce6-9b4e-4644-b5a0-92fea00a4320-cabundle-cert\") pod \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\" (UID: \"03f56ce6-9b4e-4644-b5a0-92fea00a4320\") "
Jul 20 10:41:44.636276 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.636252    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "03f56ce6-9b4e-4644-b5a0-92fea00a4320" (UID: "03f56ce6-9b4e-4644-b5a0-92fea00a4320"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:41:44.636362 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.636343    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f56ce6-9b4e-4644-b5a0-92fea00a4320-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "03f56ce6-9b4e-4644-b5a0-92fea00a4320" (UID: "03f56ce6-9b4e-4644-b5a0-92fea00a4320"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:41:44.636362 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.636344    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03f56ce6-9b4e-4644-b5a0-92fea00a4320" (UID: "03f56ce6-9b4e-4644-b5a0-92fea00a4320"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:41:44.638016 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.637993    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f56ce6-9b4e-4644-b5a0-92fea00a4320-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03f56ce6-9b4e-4644-b5a0-92fea00a4320" (UID: "03f56ce6-9b4e-4644-b5a0-92fea00a4320"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:41:44.638141 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.638059    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kube-api-access-s5rd7" (OuterVolumeSpecName: "kube-api-access-s5rd7") pod "03f56ce6-9b4e-4644-b5a0-92fea00a4320" (UID: "03f56ce6-9b4e-4644-b5a0-92fea00a4320"). InnerVolumeSpecName "kube-api-access-s5rd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:41:44.736482 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.736448    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:41:44.736482 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.736481    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03f56ce6-9b4e-4644-b5a0-92fea00a4320-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:41:44.736482 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.736492    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03f56ce6-9b4e-4644-b5a0-92fea00a4320-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:41:44.736662 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.736500    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:41:44.736662 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.736510    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5rd7\" (UniqueName: \"kubernetes.io/projected/03f56ce6-9b4e-4644-b5a0-92fea00a4320-kube-api-access-s5rd7\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:41:44.784581 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.784555    2561 generic.go:356] "Generic (PLEG): container finished" podID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerID="29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1" exitCode=0
Jul 20 10:41:44.784678 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.784621    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"
Jul 20 10:41:44.784678 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.784629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" event={"ID":"03f56ce6-9b4e-4644-b5a0-92fea00a4320","Type":"ContainerDied","Data":"29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1"}
Jul 20 10:41:44.784750 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.784698    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n" event={"ID":"03f56ce6-9b4e-4644-b5a0-92fea00a4320","Type":"ContainerDied","Data":"4b3f3058744c2b76aaa00e042776733dfe0d4584456634882aec26f03b09443d"}
Jul 20 10:41:44.784750 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.784726    2561 scope.go:117] "RemoveContainer" containerID="287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9"
Jul 20 10:41:44.794643 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.794624    2561 scope.go:117] "RemoveContainer" containerID="29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1"
Jul 20 10:41:44.801610 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.801589    2561 scope.go:117] "RemoveContainer" containerID="fc99740f1b262673d32a2232fedd1d167ae6375872bc78382e710e01a6e1c757"
Jul 20 10:41:44.804893 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.804873    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"]
Jul 20 10:41:44.808629 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.808607    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-4rc6n"]
Jul 20 10:41:44.808999 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.808984    2561 scope.go:117] "RemoveContainer" containerID="287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9"
Jul 20 10:41:44.809277 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:41:44.809256    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9\": container with ID starting with 287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9 not found: ID does not exist" containerID="287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9"
Jul 20 10:41:44.809342 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.809283    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9"} err="failed to get container status \"287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9\": rpc error: code = NotFound desc = could not find container \"287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9\": container with ID starting with 287670d8f5918743056bcd4f8412ba38636766405cef437817a83367051d76d9 not found: ID does not exist"
Jul 20 10:41:44.809342 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.809302    2561 scope.go:117] "RemoveContainer" containerID="29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1"
Jul 20 10:41:44.809559 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:41:44.809542    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1\": container with ID starting with 29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1 not found: ID does not exist" containerID="29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1"
Jul 20 10:41:44.809602 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.809565    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1"} err="failed to get container status \"29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1\": rpc error: code = NotFound desc = could not find container \"29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1\": container with ID starting with 29ee546b4fd21f9882a317f6353e84dd604ae46242e302cce2738dfc27bc0ea1 not found: ID does not exist"
Jul 20 10:41:44.809602 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.809581    2561 scope.go:117] "RemoveContainer" containerID="fc99740f1b262673d32a2232fedd1d167ae6375872bc78382e710e01a6e1c757"
Jul 20 10:41:44.809795 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:41:44.809779    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc99740f1b262673d32a2232fedd1d167ae6375872bc78382e710e01a6e1c757\": container with ID starting with fc99740f1b262673d32a2232fedd1d167ae6375872bc78382e710e01a6e1c757 not found: ID does not exist" containerID="fc99740f1b262673d32a2232fedd1d167ae6375872bc78382e710e01a6e1c757"
Jul 20 10:41:44.809834 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:44.809799    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc99740f1b262673d32a2232fedd1d167ae6375872bc78382e710e01a6e1c757"} err="failed to get container status \"fc99740f1b262673d32a2232fedd1d167ae6375872bc78382e710e01a6e1c757\": rpc error: code = NotFound desc = could not find container \"fc99740f1b262673d32a2232fedd1d167ae6375872bc78382e710e01a6e1c757\": container with ID starting with fc99740f1b262673d32a2232fedd1d167ae6375872bc78382e710e01a6e1c757 not found: ID does not exist"
Jul 20 10:41:46.504868 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.504835    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"]
Jul 20 10:41:46.505341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.505062    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kube-rbac-proxy"
Jul 20 10:41:46.505341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.505086    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kube-rbac-proxy"
Jul 20 10:41:46.505341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.505102    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="storage-initializer"
Jul 20 10:41:46.505341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.505108    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="storage-initializer"
Jul 20 10:41:46.505341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.505134    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container"
Jul 20 10:41:46.505341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.505140    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container"
Jul 20 10:41:46.505341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.505182    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kube-rbac-proxy"
Jul 20 10:41:46.505341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.505189    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" containerName="kserve-container"
Jul 20 10:41:46.509579 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.509554    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.512138 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.512094    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 20 10:41:46.512257 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.512145    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:41:46.512318 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.512307    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:41:46.512364 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.512351    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:41:46.512413 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.512351    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 20 10:41:46.512471 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.512422    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:41:46.513304 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.513284    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:41:46.519469 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.519435    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"]
Jul 20 10:41:46.550778 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.550751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp6tm\" (UniqueName: \"kubernetes.io/projected/b391986c-fdb2-4485-b14b-6ac98833f407-kube-api-access-bp6tm\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.550918 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.550788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b391986c-fdb2-4485-b14b-6ac98833f407-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.550918 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.550872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b391986c-fdb2-4485-b14b-6ac98833f407-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.550918 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.550899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b391986c-fdb2-4485-b14b-6ac98833f407-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.551067 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.550956    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b391986c-fdb2-4485-b14b-6ac98833f407-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.652157 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.652131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b391986c-fdb2-4485-b14b-6ac98833f407-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.652267 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.652172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b391986c-fdb2-4485-b14b-6ac98833f407-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.652267 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.652224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b391986c-fdb2-4485-b14b-6ac98833f407-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.652369 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.652282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6tm\" (UniqueName: \"kubernetes.io/projected/b391986c-fdb2-4485-b14b-6ac98833f407-kube-api-access-bp6tm\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.652369 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.652310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b391986c-fdb2-4485-b14b-6ac98833f407-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.652696 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.652674    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b391986c-fdb2-4485-b14b-6ac98833f407-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.652895 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.652878    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b391986c-fdb2-4485-b14b-6ac98833f407-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.652982 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.652962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b391986c-fdb2-4485-b14b-6ac98833f407-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.654497 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.654479    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b391986c-fdb2-4485-b14b-6ac98833f407-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.661261 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.661240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6tm\" (UniqueName: \"kubernetes.io/projected/b391986c-fdb2-4485-b14b-6ac98833f407-kube-api-access-bp6tm\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9922n\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.764786 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.764728    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f56ce6-9b4e-4644-b5a0-92fea00a4320" path="/var/lib/kubelet/pods/03f56ce6-9b4e-4644-b5a0-92fea00a4320/volumes"
Jul 20 10:41:46.822047 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.822026    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:46.949154 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:46.949128    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"]
Jul 20 10:41:46.951511 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:41:46.951486    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb391986c_fdb2_4485_b14b_6ac98833f407.slice/crio-1a9ce0afd62c02aa71ba547a16a7d3c21db2ffb02eac183c21bbac46123b6f66 WatchSource:0}: Error finding container 1a9ce0afd62c02aa71ba547a16a7d3c21db2ffb02eac183c21bbac46123b6f66: Status 404 returned error can't find the container with id 1a9ce0afd62c02aa71ba547a16a7d3c21db2ffb02eac183c21bbac46123b6f66
Jul 20 10:41:47.797390 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:47.797363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" event={"ID":"b391986c-fdb2-4485-b14b-6ac98833f407","Type":"ContainerStarted","Data":"3a991fc9b5b64350d2ff4a883504bbb3dce10d304daab32b627a4f7114476456"}
Jul 20 10:41:47.797742 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:47.797415    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" event={"ID":"b391986c-fdb2-4485-b14b-6ac98833f407","Type":"ContainerStarted","Data":"1a9ce0afd62c02aa71ba547a16a7d3c21db2ffb02eac183c21bbac46123b6f66"}
Jul 20 10:41:50.810557 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:50.810524    2561 generic.go:356] "Generic (PLEG): container finished" podID="b391986c-fdb2-4485-b14b-6ac98833f407" containerID="3a991fc9b5b64350d2ff4a883504bbb3dce10d304daab32b627a4f7114476456" exitCode=0
Jul 20 10:41:50.810930 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:50.810594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" event={"ID":"b391986c-fdb2-4485-b14b-6ac98833f407","Type":"ContainerDied","Data":"3a991fc9b5b64350d2ff4a883504bbb3dce10d304daab32b627a4f7114476456"}
Jul 20 10:41:51.816627 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:51.816592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" event={"ID":"b391986c-fdb2-4485-b14b-6ac98833f407","Type":"ContainerStarted","Data":"48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97"}
Jul 20 10:41:51.816994 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:51.816658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" event={"ID":"b391986c-fdb2-4485-b14b-6ac98833f407","Type":"ContainerStarted","Data":"b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b"}
Jul 20 10:41:51.816994 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:51.816854    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:51.837080 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:51.837030    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podStartSLOduration=2.176387338 podStartE2EDuration="5.837016327s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.660628989s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:41:51.836544366 +0000 UTC m=+2691.674642993" watchObservedRunningTime="2026-07-20 10:41:51.837016327 +0000 UTC m=+2691.675114912"
Jul 20 10:41:52.819959 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:52.819881    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:52.821230 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:52.821206    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 20 10:41:53.823339 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:53.823294    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 20 10:41:58.828257 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:58.828230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:41:58.828707 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:41:58.828682    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 20 10:42:08.828928 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:42:08.828894    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 20 10:42:16.119467 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:42:16.119438    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:42:16.123473 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:42:16.123454    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:42:18.829585 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:42:18.829550    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 20 10:42:28.829615 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:42:28.829572    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 20 10:42:38.829671 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:42:38.829633    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 20 10:42:48.828995 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:42:48.828954    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 20 10:42:58.829436 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:42:58.829396    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 20 10:43:08.830099 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:08.830070    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:43:17.608689 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:17.608657    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"]
Jul 20 10:43:17.609076 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:17.608886    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container" containerID="cri-o://b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b" gracePeriod=30
Jul 20 10:43:17.609076 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:17.608966    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kube-rbac-proxy" containerID="cri-o://48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97" gracePeriod=30
Jul 20 10:43:18.121499 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:18.121464    2561 generic.go:356] "Generic (PLEG): container finished" podID="b391986c-fdb2-4485-b14b-6ac98833f407" containerID="48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97" exitCode=2
Jul 20 10:43:18.121662 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:18.121507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" event={"ID":"b391986c-fdb2-4485-b14b-6ac98833f407","Type":"ContainerDied","Data":"48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97"}
Jul 20 10:43:18.824446 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:18.824414    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Jul 20 10:43:18.828728 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:18.828707    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 20 10:43:20.851004 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:20.850980    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:43:20.922882 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:20.922821    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b391986c-fdb2-4485-b14b-6ac98833f407-kube-rbac-proxy-sar-config\") pod \"b391986c-fdb2-4485-b14b-6ac98833f407\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") "
Jul 20 10:43:20.922882 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:20.922858    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b391986c-fdb2-4485-b14b-6ac98833f407-cabundle-cert\") pod \"b391986c-fdb2-4485-b14b-6ac98833f407\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") "
Jul 20 10:43:20.923070 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:20.922890    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b391986c-fdb2-4485-b14b-6ac98833f407-kserve-provision-location\") pod \"b391986c-fdb2-4485-b14b-6ac98833f407\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") "
Jul 20 10:43:20.923070 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:20.922908    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp6tm\" (UniqueName: \"kubernetes.io/projected/b391986c-fdb2-4485-b14b-6ac98833f407-kube-api-access-bp6tm\") pod \"b391986c-fdb2-4485-b14b-6ac98833f407\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") "
Jul 20 10:43:20.923070 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:20.922940    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b391986c-fdb2-4485-b14b-6ac98833f407-proxy-tls\") pod \"b391986c-fdb2-4485-b14b-6ac98833f407\" (UID: \"b391986c-fdb2-4485-b14b-6ac98833f407\") "
Jul 20 10:43:20.923262 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:20.923205    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b391986c-fdb2-4485-b14b-6ac98833f407-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b391986c-fdb2-4485-b14b-6ac98833f407" (UID: "b391986c-fdb2-4485-b14b-6ac98833f407"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:43:20.923262 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:20.923219    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b391986c-fdb2-4485-b14b-6ac98833f407-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b391986c-fdb2-4485-b14b-6ac98833f407" (UID: "b391986c-fdb2-4485-b14b-6ac98833f407"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:43:20.923262 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:20.923229    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b391986c-fdb2-4485-b14b-6ac98833f407-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b391986c-fdb2-4485-b14b-6ac98833f407" (UID: "b391986c-fdb2-4485-b14b-6ac98833f407"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:43:20.925000 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:20.924981    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b391986c-fdb2-4485-b14b-6ac98833f407-kube-api-access-bp6tm" (OuterVolumeSpecName: "kube-api-access-bp6tm") pod "b391986c-fdb2-4485-b14b-6ac98833f407" (UID: "b391986c-fdb2-4485-b14b-6ac98833f407"). InnerVolumeSpecName "kube-api-access-bp6tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:43:20.925000 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:20.924989    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b391986c-fdb2-4485-b14b-6ac98833f407-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b391986c-fdb2-4485-b14b-6ac98833f407" (UID: "b391986c-fdb2-4485-b14b-6ac98833f407"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:43:21.023715 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.023691    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b391986c-fdb2-4485-b14b-6ac98833f407-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:43:21.023715 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.023712    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp6tm\" (UniqueName: \"kubernetes.io/projected/b391986c-fdb2-4485-b14b-6ac98833f407-kube-api-access-bp6tm\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:43:21.023855 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.023722    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b391986c-fdb2-4485-b14b-6ac98833f407-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:43:21.023855 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.023732    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b391986c-fdb2-4485-b14b-6ac98833f407-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:43:21.023855 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.023741    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b391986c-fdb2-4485-b14b-6ac98833f407-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:43:21.134313 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.134282    2561 generic.go:356] "Generic (PLEG): container finished" podID="b391986c-fdb2-4485-b14b-6ac98833f407" containerID="b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b" exitCode=0
Jul 20 10:43:21.134454 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.134338    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"
Jul 20 10:43:21.134454 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.134360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" event={"ID":"b391986c-fdb2-4485-b14b-6ac98833f407","Type":"ContainerDied","Data":"b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b"}
Jul 20 10:43:21.134454 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.134424    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n" event={"ID":"b391986c-fdb2-4485-b14b-6ac98833f407","Type":"ContainerDied","Data":"1a9ce0afd62c02aa71ba547a16a7d3c21db2ffb02eac183c21bbac46123b6f66"}
Jul 20 10:43:21.134556 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.134460    2561 scope.go:117] "RemoveContainer" containerID="48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97"
Jul 20 10:43:21.142567 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.142549    2561 scope.go:117] "RemoveContainer" containerID="b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b"
Jul 20 10:43:21.149650 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.149635    2561 scope.go:117] "RemoveContainer" containerID="3a991fc9b5b64350d2ff4a883504bbb3dce10d304daab32b627a4f7114476456"
Jul 20 10:43:21.157881 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.157865    2561 scope.go:117] "RemoveContainer" containerID="48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97"
Jul 20 10:43:21.158137 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:43:21.158101    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97\": container with ID starting with 48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97 not found: ID does not exist" containerID="48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97"
Jul 20 10:43:21.158199 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.158147    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97"} err="failed to get container status \"48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97\": rpc error: code = NotFound desc = could not find container \"48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97\": container with ID starting with 48cc7915cd87f26c965de74a6421e4fc39d37c24f1c9743bb4b507c348869f97 not found: ID does not exist"
Jul 20 10:43:21.158199 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.158168    2561 scope.go:117] "RemoveContainer" containerID="b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b"
Jul 20 10:43:21.158380 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:43:21.158365    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b\": container with ID starting with b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b not found: ID does not exist" containerID="b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b"
Jul 20 10:43:21.158426 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.158383    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b"} err="failed to get container status \"b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b\": rpc error: code = NotFound desc = could not find container \"b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b\": container with ID starting with b6cbbfedde2902fb627356dc959faa31b50514e804bb6e89b5fa49bb8df6849b not found: ID does not exist"
Jul 20 10:43:21.158426 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.158395    2561 scope.go:117] "RemoveContainer" containerID="3a991fc9b5b64350d2ff4a883504bbb3dce10d304daab32b627a4f7114476456"
Jul 20 10:43:21.158657 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:43:21.158636    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a991fc9b5b64350d2ff4a883504bbb3dce10d304daab32b627a4f7114476456\": container with ID starting with 3a991fc9b5b64350d2ff4a883504bbb3dce10d304daab32b627a4f7114476456 not found: ID does not exist" containerID="3a991fc9b5b64350d2ff4a883504bbb3dce10d304daab32b627a4f7114476456"
Jul 20 10:43:21.158755 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.158660    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a991fc9b5b64350d2ff4a883504bbb3dce10d304daab32b627a4f7114476456"} err="failed to get container status \"3a991fc9b5b64350d2ff4a883504bbb3dce10d304daab32b627a4f7114476456\": rpc error: code = NotFound desc = could not find container \"3a991fc9b5b64350d2ff4a883504bbb3dce10d304daab32b627a4f7114476456\": container with ID starting with 3a991fc9b5b64350d2ff4a883504bbb3dce10d304daab32b627a4f7114476456 not found: ID does not exist"
Jul 20 10:43:21.162662 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.162641    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"]
Jul 20 10:43:21.168602 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:21.168580    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9922n"]
Jul 20 10:43:22.766050 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.765974    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" path="/var/lib/kubelet/pods/b391986c-fdb2-4485-b14b-6ac98833f407/volumes"
Jul 20 10:43:22.801835 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.801804    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"]
Jul 20 10:43:22.802054 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.802042    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="storage-initializer"
Jul 20 10:43:22.802098 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.802056    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="storage-initializer"
Jul 20 10:43:22.802098 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.802065    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kube-rbac-proxy"
Jul 20 10:43:22.802098 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.802071    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kube-rbac-proxy"
Jul 20 10:43:22.802098 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.802080    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container"
Jul 20 10:43:22.802098 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.802085    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container"
Jul 20 10:43:22.802278 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.802163    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kube-rbac-proxy"
Jul 20 10:43:22.802278 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.802170    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b391986c-fdb2-4485-b14b-6ac98833f407" containerName="kserve-container"
Jul 20 10:43:22.806593 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.806573    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.810409 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.810388    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-78c4fe-kube-rbac-proxy-sar-config\""
Jul 20 10:43:22.810523 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.810424    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:43:22.810523 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.810427    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:43:22.810892 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.810877    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-78c4fe-predictor-serving-cert\""
Jul 20 10:43:22.810993 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.810881    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:43:22.810993 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.810911    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:43:22.811224 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.811209    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:43:22.823060 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.823035    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"]
Jul 20 10:43:22.837478 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.837456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca450c63-6069-4906-8945-433efd89033c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.837618 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.837495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca450c63-6069-4906-8945-433efd89033c-cabundle-cert\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.837618 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.837528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca450c63-6069-4906-8945-433efd89033c-kserve-provision-location\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.837618 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.837569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrjnc\" (UniqueName: \"kubernetes.io/projected/ca450c63-6069-4906-8945-433efd89033c-kube-api-access-wrjnc\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.837618 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.837588    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca450c63-6069-4906-8945-433efd89033c-proxy-tls\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.938919 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.938876    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca450c63-6069-4906-8945-433efd89033c-cabundle-cert\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.941721 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.939169    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca450c63-6069-4906-8945-433efd89033c-kserve-provision-location\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.941721 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.939245    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrjnc\" (UniqueName: \"kubernetes.io/projected/ca450c63-6069-4906-8945-433efd89033c-kube-api-access-wrjnc\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.941721 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.939282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca450c63-6069-4906-8945-433efd89033c-proxy-tls\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.941721 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.939354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca450c63-6069-4906-8945-433efd89033c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.941721 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.940140    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca450c63-6069-4906-8945-433efd89033c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.941721 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.940708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca450c63-6069-4906-8945-433efd89033c-cabundle-cert\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.941721 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.940974    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca450c63-6069-4906-8945-433efd89033c-kserve-provision-location\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.944832 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.944807    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca450c63-6069-4906-8945-433efd89033c-proxy-tls\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:22.953277 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:22.953255    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrjnc\" (UniqueName: \"kubernetes.io/projected/ca450c63-6069-4906-8945-433efd89033c-kube-api-access-wrjnc\") pod \"isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") " pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:23.116472 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:23.116451    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:23.260048 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:23.260012    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"]
Jul 20 10:43:23.262851 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:43:23.262823    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca450c63_6069_4906_8945_433efd89033c.slice/crio-3733b51fde01d73e86a0eec5c3efe460a7103d3253818554009e3272ae8bdb83 WatchSource:0}: Error finding container 3733b51fde01d73e86a0eec5c3efe460a7103d3253818554009e3272ae8bdb83: Status 404 returned error can't find the container with id 3733b51fde01d73e86a0eec5c3efe460a7103d3253818554009e3272ae8bdb83
Jul 20 10:43:24.146908 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:24.146874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" event={"ID":"ca450c63-6069-4906-8945-433efd89033c","Type":"ContainerStarted","Data":"a9db5a141e9db69421e6c6c4987cdcbf9a2cf3f107b57187590d60fefd5f2bde"}
Jul 20 10:43:24.147322 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:24.146929    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" event={"ID":"ca450c63-6069-4906-8945-433efd89033c","Type":"ContainerStarted","Data":"3733b51fde01d73e86a0eec5c3efe460a7103d3253818554009e3272ae8bdb83"}
Jul 20 10:43:27.158079 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:27.158047    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca450c63-6069-4906-8945-433efd89033c" containerID="a9db5a141e9db69421e6c6c4987cdcbf9a2cf3f107b57187590d60fefd5f2bde" exitCode=0
Jul 20 10:43:27.158449 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:27.158137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" event={"ID":"ca450c63-6069-4906-8945-433efd89033c","Type":"ContainerDied","Data":"a9db5a141e9db69421e6c6c4987cdcbf9a2cf3f107b57187590d60fefd5f2bde"}
Jul 20 10:43:28.164023 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:28.163978    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" event={"ID":"ca450c63-6069-4906-8945-433efd89033c","Type":"ContainerStarted","Data":"8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d"}
Jul 20 10:43:28.164445 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:28.164034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" event={"ID":"ca450c63-6069-4906-8945-433efd89033c","Type":"ContainerStarted","Data":"fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8"}
Jul 20 10:43:28.164445 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:28.164243    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:29.167800 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:29.167769    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:29.169050 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:29.169022    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 20 10:43:30.170861 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:30.170826    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 20 10:43:35.175592 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:35.175560    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:43:35.176145 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:35.176101    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 20 10:43:35.199827 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:35.199783    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podStartSLOduration=9.495538132 podStartE2EDuration="13.199770127s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.704231995s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:43:28.194732527 +0000 UTC m=+2788.032831121" watchObservedRunningTime="2026-07-20 10:43:35.199770127 +0000 UTC m=+2795.037868712"
Jul 20 10:43:45.176283 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:45.176239    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 20 10:43:55.176274 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:43:55.176229    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 20 10:44:05.176782 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:44:05.176746    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 20 10:44:15.176668 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:44:15.176627    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 20 10:44:25.176624 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:44:25.176578    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 20 10:44:35.176950 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:44:35.176907    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 20 10:44:45.177272 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:44:45.177239    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:45:07.892960 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:07.892926    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"]
Jul 20 10:45:07.896162 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:07.896144    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:07.898599 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:07.898570    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-78c4fe-predictor-serving-cert\""
Jul 20 10:45:07.898702 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:07.898608    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-78c4fe-kube-rbac-proxy-sar-config\""
Jul 20 10:45:07.898702 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:07.898646    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-78c4fe\""
Jul 20 10:45:07.898702 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:07.898608    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-78c4fe-dockercfg-vkwlb\""
Jul 20 10:45:07.910292 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:07.910264    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"]
Jul 20 10:45:07.983781 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:07.983751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0789676c-24ca-443c-bfda-4583278d702a-proxy-tls\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:07.983938 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:07.983801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trdvq\" (UniqueName: \"kubernetes.io/projected/0789676c-24ca-443c-bfda-4583278d702a-kube-api-access-trdvq\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:07.983938 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:07.983827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0789676c-24ca-443c-bfda-4583278d702a-cabundle-cert\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:07.983938 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:07.983848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0789676c-24ca-443c-bfda-4583278d702a-kserve-provision-location\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:07.983938 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:07.983873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0789676c-24ca-443c-bfda-4583278d702a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.084428 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.084396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0789676c-24ca-443c-bfda-4583278d702a-cabundle-cert\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.084541 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.084451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0789676c-24ca-443c-bfda-4583278d702a-kserve-provision-location\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.084541 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.084497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0789676c-24ca-443c-bfda-4583278d702a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.084541 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.084529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0789676c-24ca-443c-bfda-4583278d702a-proxy-tls\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.084711 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.084586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trdvq\" (UniqueName: \"kubernetes.io/projected/0789676c-24ca-443c-bfda-4583278d702a-kube-api-access-trdvq\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.084833 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:45:08.084725    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-serving-cert: secret "isvc-secondary-78c4fe-predictor-serving-cert" not found
Jul 20 10:45:08.084833 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:45:08.084801    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0789676c-24ca-443c-bfda-4583278d702a-proxy-tls podName:0789676c-24ca-443c-bfda-4583278d702a nodeName:}" failed. No retries permitted until 2026-07-20 10:45:08.584784779 +0000 UTC m=+2888.422883345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0789676c-24ca-443c-bfda-4583278d702a-proxy-tls") pod "isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44" (UID: "0789676c-24ca-443c-bfda-4583278d702a") : secret "isvc-secondary-78c4fe-predictor-serving-cert" not found
Jul 20 10:45:08.084970 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.084919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0789676c-24ca-443c-bfda-4583278d702a-kserve-provision-location\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.085142 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.085105    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0789676c-24ca-443c-bfda-4583278d702a-cabundle-cert\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.085258 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.085241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0789676c-24ca-443c-bfda-4583278d702a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.093223 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.093202    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trdvq\" (UniqueName: \"kubernetes.io/projected/0789676c-24ca-443c-bfda-4583278d702a-kube-api-access-trdvq\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.587786 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.587743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0789676c-24ca-443c-bfda-4583278d702a-proxy-tls\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.590099 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.590080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0789676c-24ca-443c-bfda-4583278d702a-proxy-tls\") pod \"isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") " pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.806036 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.806000    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:08.932678 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:08.932650    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"]
Jul 20 10:45:08.934606 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:45:08.934575    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0789676c_24ca_443c_bfda_4583278d702a.slice/crio-6ea36284af947f54a378392a7e645e84d9809516c87b5a35223510ad95a46ca5 WatchSource:0}: Error finding container 6ea36284af947f54a378392a7e645e84d9809516c87b5a35223510ad95a46ca5: Status 404 returned error can't find the container with id 6ea36284af947f54a378392a7e645e84d9809516c87b5a35223510ad95a46ca5
Jul 20 10:45:09.522192 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:09.522161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44" event={"ID":"0789676c-24ca-443c-bfda-4583278d702a","Type":"ContainerStarted","Data":"dfeaaf4cfc95014ce1de2adb97fbd67b1671692c8ae6d1eb54106e4376413be6"}
Jul 20 10:45:09.522369 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:09.522216    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44" event={"ID":"0789676c-24ca-443c-bfda-4583278d702a","Type":"ContainerStarted","Data":"6ea36284af947f54a378392a7e645e84d9809516c87b5a35223510ad95a46ca5"}
Jul 20 10:45:13.537670 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:13.537643    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44_0789676c-24ca-443c-bfda-4583278d702a/storage-initializer/0.log"
Jul 20 10:45:13.538062 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:13.537680    2561 generic.go:356] "Generic (PLEG): container finished" podID="0789676c-24ca-443c-bfda-4583278d702a" containerID="dfeaaf4cfc95014ce1de2adb97fbd67b1671692c8ae6d1eb54106e4376413be6" exitCode=1
Jul 20 10:45:13.538062 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:13.537757    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44" event={"ID":"0789676c-24ca-443c-bfda-4583278d702a","Type":"ContainerDied","Data":"dfeaaf4cfc95014ce1de2adb97fbd67b1671692c8ae6d1eb54106e4376413be6"}
Jul 20 10:45:14.543472 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:14.543445    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44_0789676c-24ca-443c-bfda-4583278d702a/storage-initializer/0.log"
Jul 20 10:45:14.543870 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:14.543503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44" event={"ID":"0789676c-24ca-443c-bfda-4583278d702a","Type":"ContainerStarted","Data":"22c6bce0baf31dcf9d23b5573193c6bcec981a9bd976d5c68212366076af62f7"}
Jul 20 10:45:17.556402 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:17.556377    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44_0789676c-24ca-443c-bfda-4583278d702a/storage-initializer/1.log"
Jul 20 10:45:17.556795 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:17.556697    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44_0789676c-24ca-443c-bfda-4583278d702a/storage-initializer/0.log"
Jul 20 10:45:17.556795 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:17.556725    2561 generic.go:356] "Generic (PLEG): container finished" podID="0789676c-24ca-443c-bfda-4583278d702a" containerID="22c6bce0baf31dcf9d23b5573193c6bcec981a9bd976d5c68212366076af62f7" exitCode=1
Jul 20 10:45:17.556795 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:17.556761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44" event={"ID":"0789676c-24ca-443c-bfda-4583278d702a","Type":"ContainerDied","Data":"22c6bce0baf31dcf9d23b5573193c6bcec981a9bd976d5c68212366076af62f7"}
Jul 20 10:45:17.556924 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:17.556813    2561 scope.go:117] "RemoveContainer" containerID="dfeaaf4cfc95014ce1de2adb97fbd67b1671692c8ae6d1eb54106e4376413be6"
Jul 20 10:45:17.557158 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:17.557132    2561 scope.go:117] "RemoveContainer" containerID="dfeaaf4cfc95014ce1de2adb97fbd67b1671692c8ae6d1eb54106e4376413be6"
Jul 20 10:45:17.566903 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:45:17.566881    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44_kserve-ci-e2e-test(0789676c-24ca-443c-bfda-4583278d702a)\"" pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44" podUID="0789676c-24ca-443c-bfda-4583278d702a"
Jul 20 10:45:17.567139 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:45:17.567098    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44_kserve-ci-e2e-test_0789676c-24ca-443c-bfda-4583278d702a_0 in pod sandbox 6ea36284af947f54a378392a7e645e84d9809516c87b5a35223510ad95a46ca5 from index: no such id: 'dfeaaf4cfc95014ce1de2adb97fbd67b1671692c8ae6d1eb54106e4376413be6'" containerID="dfeaaf4cfc95014ce1de2adb97fbd67b1671692c8ae6d1eb54106e4376413be6"
Jul 20 10:45:17.567199 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:17.567149    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfeaaf4cfc95014ce1de2adb97fbd67b1671692c8ae6d1eb54106e4376413be6"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44_kserve-ci-e2e-test_0789676c-24ca-443c-bfda-4583278d702a_0 in pod sandbox 6ea36284af947f54a378392a7e645e84d9809516c87b5a35223510ad95a46ca5 from index: no such id: 'dfeaaf4cfc95014ce1de2adb97fbd67b1671692c8ae6d1eb54106e4376413be6'"
Jul 20 10:45:18.562029 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:18.562004    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44_0789676c-24ca-443c-bfda-4583278d702a/storage-initializer/1.log"
Jul 20 10:45:23.945879 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:23.945847    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"]
Jul 20 10:45:24.003322 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.003293    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"]
Jul 20 10:45:24.003805 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.003592    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container" containerID="cri-o://fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8" gracePeriod=30
Jul 20 10:45:24.003805 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.003648    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kube-rbac-proxy" containerID="cri-o://8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d" gracePeriod=30
Jul 20 10:45:24.121965 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.121946    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44_0789676c-24ca-443c-bfda-4583278d702a/storage-initializer/1.log"
Jul 20 10:45:24.123994 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.123979    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:24.202690 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.202609    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0789676c-24ca-443c-bfda-4583278d702a-kube-rbac-proxy-sar-config\") pod \"0789676c-24ca-443c-bfda-4583278d702a\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") "
Jul 20 10:45:24.202690 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.202649    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trdvq\" (UniqueName: \"kubernetes.io/projected/0789676c-24ca-443c-bfda-4583278d702a-kube-api-access-trdvq\") pod \"0789676c-24ca-443c-bfda-4583278d702a\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") "
Jul 20 10:45:24.202690 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.202677    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0789676c-24ca-443c-bfda-4583278d702a-cabundle-cert\") pod \"0789676c-24ca-443c-bfda-4583278d702a\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") "
Jul 20 10:45:24.202928 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.202710    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0789676c-24ca-443c-bfda-4583278d702a-proxy-tls\") pod \"0789676c-24ca-443c-bfda-4583278d702a\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") "
Jul 20 10:45:24.202928 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.202775    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0789676c-24ca-443c-bfda-4583278d702a-kserve-provision-location\") pod \"0789676c-24ca-443c-bfda-4583278d702a\" (UID: \"0789676c-24ca-443c-bfda-4583278d702a\") "
Jul 20 10:45:24.203159 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.203056    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0789676c-24ca-443c-bfda-4583278d702a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0789676c-24ca-443c-bfda-4583278d702a" (UID: "0789676c-24ca-443c-bfda-4583278d702a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:45:24.203230 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.203153    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0789676c-24ca-443c-bfda-4583278d702a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0789676c-24ca-443c-bfda-4583278d702a" (UID: "0789676c-24ca-443c-bfda-4583278d702a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:45:24.203230 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.203171    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0789676c-24ca-443c-bfda-4583278d702a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0789676c-24ca-443c-bfda-4583278d702a" (UID: "0789676c-24ca-443c-bfda-4583278d702a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:45:24.204735 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.204700    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0789676c-24ca-443c-bfda-4583278d702a-kube-api-access-trdvq" (OuterVolumeSpecName: "kube-api-access-trdvq") pod "0789676c-24ca-443c-bfda-4583278d702a" (UID: "0789676c-24ca-443c-bfda-4583278d702a"). InnerVolumeSpecName "kube-api-access-trdvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:45:24.204827 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.204736    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0789676c-24ca-443c-bfda-4583278d702a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0789676c-24ca-443c-bfda-4583278d702a" (UID: "0789676c-24ca-443c-bfda-4583278d702a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:45:24.304279 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.304237    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0789676c-24ca-443c-bfda-4583278d702a-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:24.304279 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.304274    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0789676c-24ca-443c-bfda-4583278d702a-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:24.304279 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.304286    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0789676c-24ca-443c-bfda-4583278d702a-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:24.304279 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.304294    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0789676c-24ca-443c-bfda-4583278d702a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:24.304532 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.304303    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trdvq\" (UniqueName: \"kubernetes.io/projected/0789676c-24ca-443c-bfda-4583278d702a-kube-api-access-trdvq\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:24.584667 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.584589    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44_0789676c-24ca-443c-bfda-4583278d702a/storage-initializer/1.log"
Jul 20 10:45:24.584830 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.584701    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"
Jul 20 10:45:24.584830 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.584706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44" event={"ID":"0789676c-24ca-443c-bfda-4583278d702a","Type":"ContainerDied","Data":"6ea36284af947f54a378392a7e645e84d9809516c87b5a35223510ad95a46ca5"}
Jul 20 10:45:24.584830 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.584779    2561 scope.go:117] "RemoveContainer" containerID="22c6bce0baf31dcf9d23b5573193c6bcec981a9bd976d5c68212366076af62f7"
Jul 20 10:45:24.586921 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.586878    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca450c63-6069-4906-8945-433efd89033c" containerID="8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d" exitCode=2
Jul 20 10:45:24.587034 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.586943    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" event={"ID":"ca450c63-6069-4906-8945-433efd89033c","Type":"ContainerDied","Data":"8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d"}
Jul 20 10:45:24.621451 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.621427    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"]
Jul 20 10:45:24.624902 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.624883    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-78c4fe-predictor-77b8f88db6-xkm44"]
Jul 20 10:45:24.766312 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:24.766284    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0789676c-24ca-443c-bfda-4583278d702a" path="/var/lib/kubelet/pods/0789676c-24ca-443c-bfda-4583278d702a/volumes"
Jul 20 10:45:25.171909 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:25.171871    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Jul 20 10:45:25.176234 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:25.176212    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 20 10:45:28.139454 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.139432    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:45:28.232839 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.232770    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca450c63-6069-4906-8945-433efd89033c-kube-rbac-proxy-sar-config\") pod \"ca450c63-6069-4906-8945-433efd89033c\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") "
Jul 20 10:45:28.232839 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.232798    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca450c63-6069-4906-8945-433efd89033c-cabundle-cert\") pod \"ca450c63-6069-4906-8945-433efd89033c\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") "
Jul 20 10:45:28.232839 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.232820    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca450c63-6069-4906-8945-433efd89033c-proxy-tls\") pod \"ca450c63-6069-4906-8945-433efd89033c\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") "
Jul 20 10:45:28.233040 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.232869    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrjnc\" (UniqueName: \"kubernetes.io/projected/ca450c63-6069-4906-8945-433efd89033c-kube-api-access-wrjnc\") pod \"ca450c63-6069-4906-8945-433efd89033c\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") "
Jul 20 10:45:28.233040 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.232902    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca450c63-6069-4906-8945-433efd89033c-kserve-provision-location\") pod \"ca450c63-6069-4906-8945-433efd89033c\" (UID: \"ca450c63-6069-4906-8945-433efd89033c\") "
Jul 20 10:45:28.233252 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.233142    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca450c63-6069-4906-8945-433efd89033c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca450c63-6069-4906-8945-433efd89033c" (UID: "ca450c63-6069-4906-8945-433efd89033c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:45:28.233252 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.233157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca450c63-6069-4906-8945-433efd89033c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca450c63-6069-4906-8945-433efd89033c" (UID: "ca450c63-6069-4906-8945-433efd89033c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:45:28.233346 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.233308    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca450c63-6069-4906-8945-433efd89033c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca450c63-6069-4906-8945-433efd89033c" (UID: "ca450c63-6069-4906-8945-433efd89033c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:45:28.234912 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.234892    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca450c63-6069-4906-8945-433efd89033c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca450c63-6069-4906-8945-433efd89033c" (UID: "ca450c63-6069-4906-8945-433efd89033c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:45:28.234912 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.234904    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca450c63-6069-4906-8945-433efd89033c-kube-api-access-wrjnc" (OuterVolumeSpecName: "kube-api-access-wrjnc") pod "ca450c63-6069-4906-8945-433efd89033c" (UID: "ca450c63-6069-4906-8945-433efd89033c"). InnerVolumeSpecName "kube-api-access-wrjnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:45:28.333620 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.333584    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrjnc\" (UniqueName: \"kubernetes.io/projected/ca450c63-6069-4906-8945-433efd89033c-kube-api-access-wrjnc\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:28.333620 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.333615    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca450c63-6069-4906-8945-433efd89033c-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:28.333822 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.333628    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca450c63-6069-4906-8945-433efd89033c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:28.333822 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.333643    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca450c63-6069-4906-8945-433efd89033c-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:28.333822 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.333658    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca450c63-6069-4906-8945-433efd89033c-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:28.603140 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.603052    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca450c63-6069-4906-8945-433efd89033c" containerID="fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8" exitCode=0
Jul 20 10:45:28.603140 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.603130    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"
Jul 20 10:45:28.603375 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.603149    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" event={"ID":"ca450c63-6069-4906-8945-433efd89033c","Type":"ContainerDied","Data":"fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8"}
Jul 20 10:45:28.603375 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.603221    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf" event={"ID":"ca450c63-6069-4906-8945-433efd89033c","Type":"ContainerDied","Data":"3733b51fde01d73e86a0eec5c3efe460a7103d3253818554009e3272ae8bdb83"}
Jul 20 10:45:28.603375 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.603258    2561 scope.go:117] "RemoveContainer" containerID="8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d"
Jul 20 10:45:28.610933 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.610914    2561 scope.go:117] "RemoveContainer" containerID="fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8"
Jul 20 10:45:28.617839 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.617824    2561 scope.go:117] "RemoveContainer" containerID="a9db5a141e9db69421e6c6c4987cdcbf9a2cf3f107b57187590d60fefd5f2bde"
Jul 20 10:45:28.623461 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.623440    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"]
Jul 20 10:45:28.625148 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.625134    2561 scope.go:117] "RemoveContainer" containerID="8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d"
Jul 20 10:45:28.625401 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:45:28.625381    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d\": container with ID starting with 8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d not found: ID does not exist" containerID="8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d"
Jul 20 10:45:28.625500 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.625408    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d"} err="failed to get container status \"8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d\": rpc error: code = NotFound desc = could not find container \"8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d\": container with ID starting with 8468b3c48e86a82858be761354acbad410ab71f731313bd01a8226672652c61d not found: ID does not exist"
Jul 20 10:45:28.625500 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.625425    2561 scope.go:117] "RemoveContainer" containerID="fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8"
Jul 20 10:45:28.625672 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:45:28.625655    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8\": container with ID starting with fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8 not found: ID does not exist" containerID="fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8"
Jul 20 10:45:28.625709 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.625680    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8"} err="failed to get container status \"fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8\": rpc error: code = NotFound desc = could not find container \"fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8\": container with ID starting with fc5730354f1b4f2367669ee6987ddd1b44121d167a834a47401aab602144d1a8 not found: ID does not exist"
Jul 20 10:45:28.625709 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.625696    2561 scope.go:117] "RemoveContainer" containerID="a9db5a141e9db69421e6c6c4987cdcbf9a2cf3f107b57187590d60fefd5f2bde"
Jul 20 10:45:28.625926 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:45:28.625910    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9db5a141e9db69421e6c6c4987cdcbf9a2cf3f107b57187590d60fefd5f2bde\": container with ID starting with a9db5a141e9db69421e6c6c4987cdcbf9a2cf3f107b57187590d60fefd5f2bde not found: ID does not exist" containerID="a9db5a141e9db69421e6c6c4987cdcbf9a2cf3f107b57187590d60fefd5f2bde"
Jul 20 10:45:28.625962 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.625932    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9db5a141e9db69421e6c6c4987cdcbf9a2cf3f107b57187590d60fefd5f2bde"} err="failed to get container status \"a9db5a141e9db69421e6c6c4987cdcbf9a2cf3f107b57187590d60fefd5f2bde\": rpc error: code = NotFound desc = could not find container \"a9db5a141e9db69421e6c6c4987cdcbf9a2cf3f107b57187590d60fefd5f2bde\": container with ID starting with a9db5a141e9db69421e6c6c4987cdcbf9a2cf3f107b57187590d60fefd5f2bde not found: ID does not exist"
Jul 20 10:45:28.629308 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.629288    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-78c4fe-predictor-5cb4f6644f-kzbwf"]
Jul 20 10:45:28.764768 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:28.764742    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca450c63-6069-4906-8945-433efd89033c" path="/var/lib/kubelet/pods/ca450c63-6069-4906-8945-433efd89033c/volumes"
Jul 20 10:45:29.160860 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.160821    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"]
Jul 20 10:45:29.161352 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161208    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0789676c-24ca-443c-bfda-4583278d702a" containerName="storage-initializer"
Jul 20 10:45:29.161352 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161225    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0789676c-24ca-443c-bfda-4583278d702a" containerName="storage-initializer"
Jul 20 10:45:29.161352 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161241    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container"
Jul 20 10:45:29.161352 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161250    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container"
Jul 20 10:45:29.161352 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161261    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kube-rbac-proxy"
Jul 20 10:45:29.161352 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161269    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kube-rbac-proxy"
Jul 20 10:45:29.161352 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161280    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="storage-initializer"
Jul 20 10:45:29.161352 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161287    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="storage-initializer"
Jul 20 10:45:29.161352 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161299    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0789676c-24ca-443c-bfda-4583278d702a" containerName="storage-initializer"
Jul 20 10:45:29.161352 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161308    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0789676c-24ca-443c-bfda-4583278d702a" containerName="storage-initializer"
Jul 20 10:45:29.161874 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161374    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0789676c-24ca-443c-bfda-4583278d702a" containerName="storage-initializer"
Jul 20 10:45:29.161874 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161385    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0789676c-24ca-443c-bfda-4583278d702a" containerName="storage-initializer"
Jul 20 10:45:29.161874 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161395    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kube-rbac-proxy"
Jul 20 10:45:29.161874 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.161404    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca450c63-6069-4906-8945-433efd89033c" containerName="kserve-container"
Jul 20 10:45:29.166354 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.166332    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.169081 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.169059    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e471dd-kube-rbac-proxy-sar-config\""
Jul 20 10:45:29.169217 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.169097    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e471dd-predictor-serving-cert\""
Jul 20 10:45:29.169217 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.169134    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:45:29.169341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.169225    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-e471dd\""
Jul 20 10:45:29.169981 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.169963    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-e471dd-dockercfg-dhchk\""
Jul 20 10:45:29.170089 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.169995    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:45:29.170089 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.170002    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:45:29.174586 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.174560    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"]
Jul 20 10:45:29.239186 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.239164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d711225-8ccc-4ee5-abd6-af0e8da1636f-cabundle-cert\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.239324 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.239201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtq6p\" (UniqueName: \"kubernetes.io/projected/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kube-api-access-qtq6p\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.239324 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.239220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kserve-provision-location\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.239324 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.239246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.239324 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.239296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d711225-8ccc-4ee5-abd6-af0e8da1636f-proxy-tls\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.340007 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.339975    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d711225-8ccc-4ee5-abd6-af0e8da1636f-proxy-tls\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.340202 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.340032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d711225-8ccc-4ee5-abd6-af0e8da1636f-cabundle-cert\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.340202 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.340062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtq6p\" (UniqueName: \"kubernetes.io/projected/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kube-api-access-qtq6p\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.340202 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.340179    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kserve-provision-location\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.340385 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.340217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.340656 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.340629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kserve-provision-location\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.340953 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.340931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.340995 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.340928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d711225-8ccc-4ee5-abd6-af0e8da1636f-cabundle-cert\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.342418 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.342396    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d711225-8ccc-4ee5-abd6-af0e8da1636f-proxy-tls\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.348951 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.348929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtq6p\" (UniqueName: \"kubernetes.io/projected/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kube-api-access-qtq6p\") pod \"isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.478247 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.478179    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:29.607786 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:29.607763    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"]
Jul 20 10:45:29.609838 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:45:29.609809    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d711225_8ccc_4ee5_abd6_af0e8da1636f.slice/crio-4a1c047a3ee7e20d22102d395c8c835996ef08a189df13e9b9371d778d38af21 WatchSource:0}: Error finding container 4a1c047a3ee7e20d22102d395c8c835996ef08a189df13e9b9371d778d38af21: Status 404 returned error can't find the container with id 4a1c047a3ee7e20d22102d395c8c835996ef08a189df13e9b9371d778d38af21
Jul 20 10:45:30.614037 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:30.614000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx" event={"ID":"5d711225-8ccc-4ee5-abd6-af0e8da1636f","Type":"ContainerStarted","Data":"abb8df211b77bf8f44d7c7b92d234d30f6c0cf54560ed70a1b2d2a3dc740540f"}
Jul 20 10:45:30.614464 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:30.614103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx" event={"ID":"5d711225-8ccc-4ee5-abd6-af0e8da1636f","Type":"ContainerStarted","Data":"4a1c047a3ee7e20d22102d395c8c835996ef08a189df13e9b9371d778d38af21"}
Jul 20 10:45:37.640682 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:37.640654    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx_5d711225-8ccc-4ee5-abd6-af0e8da1636f/storage-initializer/0.log"
Jul 20 10:45:37.641033 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:37.640692    2561 generic.go:356] "Generic (PLEG): container finished" podID="5d711225-8ccc-4ee5-abd6-af0e8da1636f" containerID="abb8df211b77bf8f44d7c7b92d234d30f6c0cf54560ed70a1b2d2a3dc740540f" exitCode=1
Jul 20 10:45:37.641033 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:37.640756    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx" event={"ID":"5d711225-8ccc-4ee5-abd6-af0e8da1636f","Type":"ContainerDied","Data":"abb8df211b77bf8f44d7c7b92d234d30f6c0cf54560ed70a1b2d2a3dc740540f"}
Jul 20 10:45:38.645575 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:38.645546    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx_5d711225-8ccc-4ee5-abd6-af0e8da1636f/storage-initializer/0.log"
Jul 20 10:45:38.645948 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:38.645649    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx" event={"ID":"5d711225-8ccc-4ee5-abd6-af0e8da1636f","Type":"ContainerStarted","Data":"c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9"}
Jul 20 10:45:39.193511 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:39.193465    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"]
Jul 20 10:45:39.649457 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:39.649415    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx" podUID="5d711225-8ccc-4ee5-abd6-af0e8da1636f" containerName="storage-initializer" containerID="cri-o://c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9" gracePeriod=30
Jul 20 10:45:44.285811 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.285787    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx_5d711225-8ccc-4ee5-abd6-af0e8da1636f/storage-initializer/1.log"
Jul 20 10:45:44.286259 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.286241    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx_5d711225-8ccc-4ee5-abd6-af0e8da1636f/storage-initializer/0.log"
Jul 20 10:45:44.288555 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.288537    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:44.348484 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.348453    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtq6p\" (UniqueName: \"kubernetes.io/projected/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kube-api-access-qtq6p\") pod \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") "
Jul 20 10:45:44.348630 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.348498    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kube-rbac-proxy-sar-config\") pod \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") "
Jul 20 10:45:44.348630 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.348520    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d711225-8ccc-4ee5-abd6-af0e8da1636f-proxy-tls\") pod \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") "
Jul 20 10:45:44.348764 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.348642    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kserve-provision-location\") pod \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") "
Jul 20 10:45:44.348764 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.348703    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d711225-8ccc-4ee5-abd6-af0e8da1636f-cabundle-cert\") pod \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\" (UID: \"5d711225-8ccc-4ee5-abd6-af0e8da1636f\") "
Jul 20 10:45:44.348926 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.348904    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5d711225-8ccc-4ee5-abd6-af0e8da1636f" (UID: "5d711225-8ccc-4ee5-abd6-af0e8da1636f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:45:44.348983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.348902    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d711225-8ccc-4ee5-abd6-af0e8da1636f" (UID: "5d711225-8ccc-4ee5-abd6-af0e8da1636f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:45:44.348983 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.348970    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d711225-8ccc-4ee5-abd6-af0e8da1636f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d711225-8ccc-4ee5-abd6-af0e8da1636f" (UID: "5d711225-8ccc-4ee5-abd6-af0e8da1636f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:45:44.351045 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.351027    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d711225-8ccc-4ee5-abd6-af0e8da1636f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d711225-8ccc-4ee5-abd6-af0e8da1636f" (UID: "5d711225-8ccc-4ee5-abd6-af0e8da1636f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:45:44.351108 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.351049    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kube-api-access-qtq6p" (OuterVolumeSpecName: "kube-api-access-qtq6p") pod "5d711225-8ccc-4ee5-abd6-af0e8da1636f" (UID: "5d711225-8ccc-4ee5-abd6-af0e8da1636f"). InnerVolumeSpecName "kube-api-access-qtq6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:45:44.450129 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.450079    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:44.450129 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.450104    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d711225-8ccc-4ee5-abd6-af0e8da1636f-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:44.450343 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.450137    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtq6p\" (UniqueName: \"kubernetes.io/projected/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kube-api-access-qtq6p\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:44.450343 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.450146    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d711225-8ccc-4ee5-abd6-af0e8da1636f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:44.450343 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.450155    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d711225-8ccc-4ee5-abd6-af0e8da1636f-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:45:44.486402 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.486364    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"]
Jul 20 10:45:44.486654 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.486641    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d711225-8ccc-4ee5-abd6-af0e8da1636f" containerName="storage-initializer"
Jul 20 10:45:44.486703 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.486656    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d711225-8ccc-4ee5-abd6-af0e8da1636f" containerName="storage-initializer"
Jul 20 10:45:44.486703 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.486666    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d711225-8ccc-4ee5-abd6-af0e8da1636f" containerName="storage-initializer"
Jul 20 10:45:44.486703 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.486672    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d711225-8ccc-4ee5-abd6-af0e8da1636f" containerName="storage-initializer"
Jul 20 10:45:44.488153 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.487449    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d711225-8ccc-4ee5-abd6-af0e8da1636f" containerName="storage-initializer"
Jul 20 10:45:44.488153 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.487476    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d711225-8ccc-4ee5-abd6-af0e8da1636f" containerName="storage-initializer"
Jul 20 10:45:44.493582 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.493559    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.496148 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.496124    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:45:44.496148 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.496140    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 20 10:45:44.496320 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.496155    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:45:44.496320 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.496300    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 20 10:45:44.501327 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.501307    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"]
Jul 20 10:45:44.550847 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.550824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0202d585-d2a7-427a-b166-0aa8cfb1af82-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.550982 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.550854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0202d585-d2a7-427a-b166-0aa8cfb1af82-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.550982 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.550871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0202d585-d2a7-427a-b166-0aa8cfb1af82-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.550982 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.550886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0202d585-d2a7-427a-b166-0aa8cfb1af82-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.551169 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.551017    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vchp8\" (UniqueName: \"kubernetes.io/projected/0202d585-d2a7-427a-b166-0aa8cfb1af82-kube-api-access-vchp8\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.651866 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.651831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vchp8\" (UniqueName: \"kubernetes.io/projected/0202d585-d2a7-427a-b166-0aa8cfb1af82-kube-api-access-vchp8\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.652034 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.651881    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0202d585-d2a7-427a-b166-0aa8cfb1af82-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.652034 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.651910    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0202d585-d2a7-427a-b166-0aa8cfb1af82-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.652034 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.651931    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0202d585-d2a7-427a-b166-0aa8cfb1af82-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.652034 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.651953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0202d585-d2a7-427a-b166-0aa8cfb1af82-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.652283 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:45:44.652071    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Jul 20 10:45:44.652283 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:45:44.652168    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0202d585-d2a7-427a-b166-0aa8cfb1af82-proxy-tls podName:0202d585-d2a7-427a-b166-0aa8cfb1af82 nodeName:}" failed. No retries permitted until 2026-07-20 10:45:45.152151385 +0000 UTC m=+2924.990249949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0202d585-d2a7-427a-b166-0aa8cfb1af82-proxy-tls") pod "isvc-predictive-sklearn-predictor-5f849d69df-bccx4" (UID: "0202d585-d2a7-427a-b166-0aa8cfb1af82") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Jul 20 10:45:44.652449 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.652425    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0202d585-d2a7-427a-b166-0aa8cfb1af82-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.652659 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.652641    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0202d585-d2a7-427a-b166-0aa8cfb1af82-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.652749 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.652727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0202d585-d2a7-427a-b166-0aa8cfb1af82-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.660930 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.660911    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vchp8\" (UniqueName: \"kubernetes.io/projected/0202d585-d2a7-427a-b166-0aa8cfb1af82-kube-api-access-vchp8\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:44.669444 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.669426    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx_5d711225-8ccc-4ee5-abd6-af0e8da1636f/storage-initializer/1.log"
Jul 20 10:45:44.669794 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.669780    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx_5d711225-8ccc-4ee5-abd6-af0e8da1636f/storage-initializer/0.log"
Jul 20 10:45:44.669861 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.669817    2561 generic.go:356] "Generic (PLEG): container finished" podID="5d711225-8ccc-4ee5-abd6-af0e8da1636f" containerID="c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9" exitCode=1
Jul 20 10:45:44.669900 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.669863    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx" event={"ID":"5d711225-8ccc-4ee5-abd6-af0e8da1636f","Type":"ContainerDied","Data":"c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9"}
Jul 20 10:45:44.669900 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.669876    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"
Jul 20 10:45:44.669965 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.669924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx" event={"ID":"5d711225-8ccc-4ee5-abd6-af0e8da1636f","Type":"ContainerDied","Data":"4a1c047a3ee7e20d22102d395c8c835996ef08a189df13e9b9371d778d38af21"}
Jul 20 10:45:44.670001 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.669962    2561 scope.go:117] "RemoveContainer" containerID="c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9"
Jul 20 10:45:44.677711 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.677692    2561 scope.go:117] "RemoveContainer" containerID="abb8df211b77bf8f44d7c7b92d234d30f6c0cf54560ed70a1b2d2a3dc740540f"
Jul 20 10:45:44.684797 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.684780    2561 scope.go:117] "RemoveContainer" containerID="c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9"
Jul 20 10:45:44.685043 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:45:44.685026    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9\": container with ID starting with c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9 not found: ID does not exist" containerID="c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9"
Jul 20 10:45:44.685088 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.685051    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9"} err="failed to get container status \"c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9\": rpc error: code = NotFound desc = could not find container \"c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9\": container with ID starting with c398c7da016153aba50f9e047a63b8e945c89135124719eb650695de4a28bff9 not found: ID does not exist"
Jul 20 10:45:44.685088 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.685069    2561 scope.go:117] "RemoveContainer" containerID="abb8df211b77bf8f44d7c7b92d234d30f6c0cf54560ed70a1b2d2a3dc740540f"
Jul 20 10:45:44.685294 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:45:44.685280    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb8df211b77bf8f44d7c7b92d234d30f6c0cf54560ed70a1b2d2a3dc740540f\": container with ID starting with abb8df211b77bf8f44d7c7b92d234d30f6c0cf54560ed70a1b2d2a3dc740540f not found: ID does not exist" containerID="abb8df211b77bf8f44d7c7b92d234d30f6c0cf54560ed70a1b2d2a3dc740540f"
Jul 20 10:45:44.685335 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.685297    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb8df211b77bf8f44d7c7b92d234d30f6c0cf54560ed70a1b2d2a3dc740540f"} err="failed to get container status \"abb8df211b77bf8f44d7c7b92d234d30f6c0cf54560ed70a1b2d2a3dc740540f\": rpc error: code = NotFound desc = could not find container \"abb8df211b77bf8f44d7c7b92d234d30f6c0cf54560ed70a1b2d2a3dc740540f\": container with ID starting with abb8df211b77bf8f44d7c7b92d234d30f6c0cf54560ed70a1b2d2a3dc740540f not found: ID does not exist"
Jul 20 10:45:44.708881 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.708829    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"]
Jul 20 10:45:44.711846 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.711823    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e471dd-predictor-6d77fc9ddb-zvtzx"]
Jul 20 10:45:44.764979 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:44.764954    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d711225-8ccc-4ee5-abd6-af0e8da1636f" path="/var/lib/kubelet/pods/5d711225-8ccc-4ee5-abd6-af0e8da1636f/volumes"
Jul 20 10:45:45.156014 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:45.155986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0202d585-d2a7-427a-b166-0aa8cfb1af82-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:45.158459 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:45.158428    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0202d585-d2a7-427a-b166-0aa8cfb1af82-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bccx4\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:45.404484 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:45.404453    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:45:45.533238 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:45.533212    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"]
Jul 20 10:45:45.535771 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:45:45.535728    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0202d585_d2a7_427a_b166_0aa8cfb1af82.slice/crio-36a74874dcf3075410e75b17bb20131d61792c5397abf011d32ec95218124fdf WatchSource:0}: Error finding container 36a74874dcf3075410e75b17bb20131d61792c5397abf011d32ec95218124fdf: Status 404 returned error can't find the container with id 36a74874dcf3075410e75b17bb20131d61792c5397abf011d32ec95218124fdf
Jul 20 10:45:45.676376 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:45.676292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" event={"ID":"0202d585-d2a7-427a-b166-0aa8cfb1af82","Type":"ContainerStarted","Data":"335bc50be8afe9a81812c99e32036380d8720218d73076066404a405b370c83a"}
Jul 20 10:45:45.676376 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:45.676345    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" event={"ID":"0202d585-d2a7-427a-b166-0aa8cfb1af82","Type":"ContainerStarted","Data":"36a74874dcf3075410e75b17bb20131d61792c5397abf011d32ec95218124fdf"}
Jul 20 10:45:49.693424 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:49.693339    2561 generic.go:356] "Generic (PLEG): container finished" podID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerID="335bc50be8afe9a81812c99e32036380d8720218d73076066404a405b370c83a" exitCode=0
Jul 20 10:45:49.693424 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:49.693413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" event={"ID":"0202d585-d2a7-427a-b166-0aa8cfb1af82","Type":"ContainerDied","Data":"335bc50be8afe9a81812c99e32036380d8720218d73076066404a405b370c83a"}
Jul 20 10:45:49.694613 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:45:49.694597    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:46:10.800102 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:10.800066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" event={"ID":"0202d585-d2a7-427a-b166-0aa8cfb1af82","Type":"ContainerStarted","Data":"2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f"}
Jul 20 10:46:10.800574 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:10.800150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" event={"ID":"0202d585-d2a7-427a-b166-0aa8cfb1af82","Type":"ContainerStarted","Data":"baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13"}
Jul 20 10:46:10.800574 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:10.800332    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:46:10.822541 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:10.822500    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podStartSLOduration=2.905406465 podStartE2EDuration="26.822486176s" totalImagesPullingTime="20.087017687s" totalInitContainerRuntime="3.830062024s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:46:10.820379258 +0000 UTC m=+2950.658477893" watchObservedRunningTime="2026-07-20 10:46:10.822486176 +0000 UTC m=+2950.660584761"
Jul 20 10:46:11.803491 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:11.803460    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:46:11.804752 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:11.804723    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 20 10:46:12.806809 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:12.806771    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 20 10:46:17.811274 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:17.811246    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:46:17.811747 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:17.811721    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 20 10:46:27.812521 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:27.812429    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 20 10:46:37.812572 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:37.812536    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 20 10:46:47.812000 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:47.811960    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 20 10:46:57.811785 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:46:57.811745    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 20 10:47:07.812334 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:07.812285    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 20 10:47:16.141912 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:16.141879    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:47:16.146897 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:16.146878    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:47:17.812691 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:17.812651    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 20 10:47:27.812261 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:27.812231    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:47:34.586429 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:34.586397    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"]
Jul 20 10:47:34.586895 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:34.586708    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container" containerID="cri-o://baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13" gracePeriod=30
Jul 20 10:47:34.586895 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:34.586736    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kube-rbac-proxy" containerID="cri-o://2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f" gracePeriod=30
Jul 20 10:47:35.092854 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:35.092826    2561 generic.go:356] "Generic (PLEG): container finished" podID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerID="2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f" exitCode=2
Jul 20 10:47:35.093024 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:35.092910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" event={"ID":"0202d585-d2a7-427a-b166-0aa8cfb1af82","Type":"ContainerDied","Data":"2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f"}
Jul 20 10:47:37.808065 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:37.808025    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Jul 20 10:47:37.812360 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:37.812332    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 20 10:47:38.720891 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.720868    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:47:38.875020 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.874991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0202d585-d2a7-427a-b166-0aa8cfb1af82-kube-rbac-proxy-sar-config\") pod \"0202d585-d2a7-427a-b166-0aa8cfb1af82\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") "
Jul 20 10:47:38.875542 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.875028    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vchp8\" (UniqueName: \"kubernetes.io/projected/0202d585-d2a7-427a-b166-0aa8cfb1af82-kube-api-access-vchp8\") pod \"0202d585-d2a7-427a-b166-0aa8cfb1af82\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") "
Jul 20 10:47:38.875542 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.875082    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0202d585-d2a7-427a-b166-0aa8cfb1af82-proxy-tls\") pod \"0202d585-d2a7-427a-b166-0aa8cfb1af82\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") "
Jul 20 10:47:38.875542 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.875176    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0202d585-d2a7-427a-b166-0aa8cfb1af82-cabundle-cert\") pod \"0202d585-d2a7-427a-b166-0aa8cfb1af82\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") "
Jul 20 10:47:38.875542 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.875221    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0202d585-d2a7-427a-b166-0aa8cfb1af82-kserve-provision-location\") pod \"0202d585-d2a7-427a-b166-0aa8cfb1af82\" (UID: \"0202d585-d2a7-427a-b166-0aa8cfb1af82\") "
Jul 20 10:47:38.875763 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.875538    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0202d585-d2a7-427a-b166-0aa8cfb1af82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0202d585-d2a7-427a-b166-0aa8cfb1af82" (UID: "0202d585-d2a7-427a-b166-0aa8cfb1af82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:47:38.875763 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.875571    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0202d585-d2a7-427a-b166-0aa8cfb1af82-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0202d585-d2a7-427a-b166-0aa8cfb1af82" (UID: "0202d585-d2a7-427a-b166-0aa8cfb1af82"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:47:38.875907 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.875885    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0202d585-d2a7-427a-b166-0aa8cfb1af82-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0202d585-d2a7-427a-b166-0aa8cfb1af82" (UID: "0202d585-d2a7-427a-b166-0aa8cfb1af82"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:47:38.877271 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.877249    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0202d585-d2a7-427a-b166-0aa8cfb1af82-kube-api-access-vchp8" (OuterVolumeSpecName: "kube-api-access-vchp8") pod "0202d585-d2a7-427a-b166-0aa8cfb1af82" (UID: "0202d585-d2a7-427a-b166-0aa8cfb1af82"). InnerVolumeSpecName "kube-api-access-vchp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:47:38.877271 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.877257    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0202d585-d2a7-427a-b166-0aa8cfb1af82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0202d585-d2a7-427a-b166-0aa8cfb1af82" (UID: "0202d585-d2a7-427a-b166-0aa8cfb1af82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:47:38.976181 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.976155    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0202d585-d2a7-427a-b166-0aa8cfb1af82-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:47:38.976181 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.976177    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0202d585-d2a7-427a-b166-0aa8cfb1af82-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:47:38.976329 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.976186    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vchp8\" (UniqueName: \"kubernetes.io/projected/0202d585-d2a7-427a-b166-0aa8cfb1af82-kube-api-access-vchp8\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:47:38.976329 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.976195    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0202d585-d2a7-427a-b166-0aa8cfb1af82-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:47:38.976329 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:38.976206    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0202d585-d2a7-427a-b166-0aa8cfb1af82-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:47:39.109394 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.109366    2561 generic.go:356] "Generic (PLEG): container finished" podID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerID="baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13" exitCode=0
Jul 20 10:47:39.109510 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.109400    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" event={"ID":"0202d585-d2a7-427a-b166-0aa8cfb1af82","Type":"ContainerDied","Data":"baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13"}
Jul 20 10:47:39.109510 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.109421    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"
Jul 20 10:47:39.109510 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.109470    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4" event={"ID":"0202d585-d2a7-427a-b166-0aa8cfb1af82","Type":"ContainerDied","Data":"36a74874dcf3075410e75b17bb20131d61792c5397abf011d32ec95218124fdf"}
Jul 20 10:47:39.109510 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.109502    2561 scope.go:117] "RemoveContainer" containerID="2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f"
Jul 20 10:47:39.118835 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.118816    2561 scope.go:117] "RemoveContainer" containerID="baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13"
Jul 20 10:47:39.127994 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.127983    2561 scope.go:117] "RemoveContainer" containerID="335bc50be8afe9a81812c99e32036380d8720218d73076066404a405b370c83a"
Jul 20 10:47:39.135037 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.135018    2561 scope.go:117] "RemoveContainer" containerID="2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f"
Jul 20 10:47:39.135332 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:47:39.135314    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f\": container with ID starting with 2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f not found: ID does not exist" containerID="2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f"
Jul 20 10:47:39.135399 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.135339    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f"} err="failed to get container status \"2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f\": rpc error: code = NotFound desc = could not find container \"2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f\": container with ID starting with 2e1c91f3b319ff88a0ded897e37639438d531d920bde47b63ed493372e600d9f not found: ID does not exist"
Jul 20 10:47:39.135399 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.135357    2561 scope.go:117] "RemoveContainer" containerID="baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13"
Jul 20 10:47:39.135579 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.135563    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"]
Jul 20 10:47:39.135616 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:47:39.135574    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13\": container with ID starting with baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13 not found: ID does not exist" containerID="baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13"
Jul 20 10:47:39.135616 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.135592    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13"} err="failed to get container status \"baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13\": rpc error: code = NotFound desc = could not find container \"baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13\": container with ID starting with baaa7064b4a019bbaa09eed3847131e4189b4e3b938023c8346e441f66debf13 not found: ID does not exist"
Jul 20 10:47:39.135616 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.135606    2561 scope.go:117] "RemoveContainer" containerID="335bc50be8afe9a81812c99e32036380d8720218d73076066404a405b370c83a"
Jul 20 10:47:39.135819 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:47:39.135805    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335bc50be8afe9a81812c99e32036380d8720218d73076066404a405b370c83a\": container with ID starting with 335bc50be8afe9a81812c99e32036380d8720218d73076066404a405b370c83a not found: ID does not exist" containerID="335bc50be8afe9a81812c99e32036380d8720218d73076066404a405b370c83a"
Jul 20 10:47:39.135857 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.135823    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335bc50be8afe9a81812c99e32036380d8720218d73076066404a405b370c83a"} err="failed to get container status \"335bc50be8afe9a81812c99e32036380d8720218d73076066404a405b370c83a\": rpc error: code = NotFound desc = could not find container \"335bc50be8afe9a81812c99e32036380d8720218d73076066404a405b370c83a\": container with ID starting with 335bc50be8afe9a81812c99e32036380d8720218d73076066404a405b370c83a not found: ID does not exist"
Jul 20 10:47:39.141150 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.141131    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bccx4"]
Jul 20 10:47:39.736271 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.736238    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"]
Jul 20 10:47:39.736624 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.736608    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kube-rbac-proxy"
Jul 20 10:47:39.736696 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.736629    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kube-rbac-proxy"
Jul 20 10:47:39.736696 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.736650    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container"
Jul 20 10:47:39.736696 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.736659    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container"
Jul 20 10:47:39.736696 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.736669    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="storage-initializer"
Jul 20 10:47:39.736696 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.736679    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="storage-initializer"
Jul 20 10:47:39.736867 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.736754    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kserve-container"
Jul 20 10:47:39.736867 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.736768    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" containerName="kube-rbac-proxy"
Jul 20 10:47:39.740002 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.739978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.742695 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.742651    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:47:39.742837 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.742651    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 20 10:47:39.742837 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.742653    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:47:39.742837 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.742653    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:47:39.743766 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.743735    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:47:39.743893 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.743846    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:47:39.743893 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.743858    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 20 10:47:39.750288 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.750270    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"]
Jul 20 10:47:39.882335 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.882303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.882335 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.882339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.882765 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.882367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.882765 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.882459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.882765 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.882496    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc48d\" (UniqueName: \"kubernetes.io/projected/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kube-api-access-hc48d\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.983257 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.983230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.983400 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.983264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.983400 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.983344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.983400 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:47:39.983365    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 20 10:47:39.983400 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.983382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.983400 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.983401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc48d\" (UniqueName: \"kubernetes.io/projected/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kube-api-access-hc48d\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.983634 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:47:39.983424    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-proxy-tls podName:e4b6eb1b-51a4-4b44-9a60-b3c3478e914c nodeName:}" failed. No retries permitted until 2026-07-20 10:47:40.483407769 +0000 UTC m=+3040.321506332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-pmg5x" (UID: "e4b6eb1b-51a4-4b44-9a60-b3c3478e914c") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 20 10:47:39.983799 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.983780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.984002 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.983987    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.984041 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.983985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:39.992565 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:39.992512    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc48d\" (UniqueName: \"kubernetes.io/projected/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kube-api-access-hc48d\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:40.487872 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:40.487838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:40.490343 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:40.490318    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-pmg5x\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:40.651247 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:40.651211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:40.765258 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:40.765190    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0202d585-d2a7-427a-b166-0aa8cfb1af82" path="/var/lib/kubelet/pods/0202d585-d2a7-427a-b166-0aa8cfb1af82/volumes"
Jul 20 10:47:40.775200 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:40.775174    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"]
Jul 20 10:47:40.777996 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:47:40.777974    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b6eb1b_51a4_4b44_9a60_b3c3478e914c.slice/crio-71cc684caf025a430c1c380b3711d08f1acc0acdc7428312fb0ce0bf86f730b2 WatchSource:0}: Error finding container 71cc684caf025a430c1c380b3711d08f1acc0acdc7428312fb0ce0bf86f730b2: Status 404 returned error can't find the container with id 71cc684caf025a430c1c380b3711d08f1acc0acdc7428312fb0ce0bf86f730b2
Jul 20 10:47:41.119784 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:41.119748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" event={"ID":"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c","Type":"ContainerStarted","Data":"d2b720a6f46ad342fed6404f83b3f1afb8a32245764be34b4d4a12b964b34e58"}
Jul 20 10:47:41.120212 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:41.119823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" event={"ID":"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c","Type":"ContainerStarted","Data":"71cc684caf025a430c1c380b3711d08f1acc0acdc7428312fb0ce0bf86f730b2"}
Jul 20 10:47:45.136716 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:45.136683    2561 generic.go:356] "Generic (PLEG): container finished" podID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerID="d2b720a6f46ad342fed6404f83b3f1afb8a32245764be34b4d4a12b964b34e58" exitCode=0
Jul 20 10:47:45.137083 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:45.136748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" event={"ID":"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c","Type":"ContainerDied","Data":"d2b720a6f46ad342fed6404f83b3f1afb8a32245764be34b4d4a12b964b34e58"}
Jul 20 10:47:46.141905 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:46.141870    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" event={"ID":"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c","Type":"ContainerStarted","Data":"317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5"}
Jul 20 10:47:46.142304 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:46.141932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" event={"ID":"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c","Type":"ContainerStarted","Data":"4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459"}
Jul 20 10:47:46.142304 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:46.142151    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:46.162576 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:46.162533    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podStartSLOduration=3.529153633 podStartE2EDuration="7.162515925s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.633362292s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:47:46.161442963 +0000 UTC m=+3045.999541579" watchObservedRunningTime="2026-07-20 10:47:46.162515925 +0000 UTC m=+3046.000614510"
Jul 20 10:47:47.145329 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:47.145294    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:47.146571 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:47.146544    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 20 10:47:48.148505 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:48.148469    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 20 10:47:53.153552 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:53.153479    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:47:53.153988 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:47:53.153957    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 20 10:48:03.154658 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:48:03.154612    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 20 10:48:13.154940 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:48:13.154899    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 20 10:48:23.154714 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:48:23.154671    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 20 10:48:33.154561 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:48:33.154519    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 20 10:48:43.154452 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:48:43.154409    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 20 10:48:53.154225 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:48:53.154183    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 20 10:49:00.766332 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:00.766299    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:49:09.892301 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:09.892269    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"]
Jul 20 10:49:09.892683 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:09.892529    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container" containerID="cri-o://4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459" gracePeriod=30
Jul 20 10:49:09.892683 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:09.892552    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kube-rbac-proxy" containerID="cri-o://317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5" gracePeriod=30
Jul 20 10:49:10.434292 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:10.434263    2561 generic.go:356] "Generic (PLEG): container finished" podID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerID="317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5" exitCode=2
Jul 20 10:49:10.434479 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:10.434318    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" event={"ID":"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c","Type":"ContainerDied","Data":"317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5"}
Jul 20 10:49:10.763461 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:10.763367    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 20 10:49:13.149375 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:13.149330    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Jul 20 10:49:14.142478 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.142457    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:49:14.244584 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.244518    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kserve-provision-location\") pod \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") "
Jul 20 10:49:14.244584 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.244556    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kube-rbac-proxy-sar-config\") pod \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") "
Jul 20 10:49:14.244584 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.244573    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc48d\" (UniqueName: \"kubernetes.io/projected/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kube-api-access-hc48d\") pod \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") "
Jul 20 10:49:14.245050 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.244595    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-proxy-tls\") pod \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") "
Jul 20 10:49:14.245050 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.244618    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-cabundle-cert\") pod \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\" (UID: \"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c\") "
Jul 20 10:49:14.245050 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.244906    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" (UID: "e4b6eb1b-51a4-4b44-9a60-b3c3478e914c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:49:14.245050 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.244937    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" (UID: "e4b6eb1b-51a4-4b44-9a60-b3c3478e914c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:49:14.245050 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.244973    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" (UID: "e4b6eb1b-51a4-4b44-9a60-b3c3478e914c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:49:14.246643 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.246621    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kube-api-access-hc48d" (OuterVolumeSpecName: "kube-api-access-hc48d") pod "e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" (UID: "e4b6eb1b-51a4-4b44-9a60-b3c3478e914c"). InnerVolumeSpecName "kube-api-access-hc48d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:49:14.246694 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.246656    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" (UID: "e4b6eb1b-51a4-4b44-9a60-b3c3478e914c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:49:14.345840 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.345803    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:49:14.345840 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.345832    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:49:14.345840 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.345841    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc48d\" (UniqueName: \"kubernetes.io/projected/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-kube-api-access-hc48d\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:49:14.346062 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.345851    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:49:14.346062 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.345861    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:49:14.449657 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.449626    2561 generic.go:356] "Generic (PLEG): container finished" podID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerID="4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459" exitCode=0
Jul 20 10:49:14.449826 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.449661    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" event={"ID":"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c","Type":"ContainerDied","Data":"4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459"}
Jul 20 10:49:14.449826 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.449684    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"
Jul 20 10:49:14.449826 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.449722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x" event={"ID":"e4b6eb1b-51a4-4b44-9a60-b3c3478e914c","Type":"ContainerDied","Data":"71cc684caf025a430c1c380b3711d08f1acc0acdc7428312fb0ce0bf86f730b2"}
Jul 20 10:49:14.449826 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.449758    2561 scope.go:117] "RemoveContainer" containerID="317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5"
Jul 20 10:49:14.460917 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.460900    2561 scope.go:117] "RemoveContainer" containerID="4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459"
Jul 20 10:49:14.467783 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.467765    2561 scope.go:117] "RemoveContainer" containerID="d2b720a6f46ad342fed6404f83b3f1afb8a32245764be34b4d4a12b964b34e58"
Jul 20 10:49:14.474886 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.474861    2561 scope.go:117] "RemoveContainer" containerID="317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5"
Jul 20 10:49:14.475171 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:49:14.475151    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5\": container with ID starting with 317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5 not found: ID does not exist" containerID="317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5"
Jul 20 10:49:14.475269 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.475179    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5"} err="failed to get container status \"317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5\": rpc error: code = NotFound desc = could not find container \"317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5\": container with ID starting with 317ff7cf6ebdee7f9e3d7dc8a39ae24b9b4d9848893e67acdac4422e22b931d5 not found: ID does not exist"
Jul 20 10:49:14.475269 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.475196    2561 scope.go:117] "RemoveContainer" containerID="4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459"
Jul 20 10:49:14.475460 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:49:14.475441    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459\": container with ID starting with 4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459 not found: ID does not exist" containerID="4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459"
Jul 20 10:49:14.475525 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.475474    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459"} err="failed to get container status \"4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459\": rpc error: code = NotFound desc = could not find container \"4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459\": container with ID starting with 4ef36b31b36b90e52fb9289e72028f1466c86806b26111e3124a1bf6c2f67459 not found: ID does not exist"
Jul 20 10:49:14.475525 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.475500    2561 scope.go:117] "RemoveContainer" containerID="d2b720a6f46ad342fed6404f83b3f1afb8a32245764be34b4d4a12b964b34e58"
Jul 20 10:49:14.475736 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:49:14.475718    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b720a6f46ad342fed6404f83b3f1afb8a32245764be34b4d4a12b964b34e58\": container with ID starting with d2b720a6f46ad342fed6404f83b3f1afb8a32245764be34b4d4a12b964b34e58 not found: ID does not exist" containerID="d2b720a6f46ad342fed6404f83b3f1afb8a32245764be34b4d4a12b964b34e58"
Jul 20 10:49:14.475791 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.475751    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b720a6f46ad342fed6404f83b3f1afb8a32245764be34b4d4a12b964b34e58"} err="failed to get container status \"d2b720a6f46ad342fed6404f83b3f1afb8a32245764be34b4d4a12b964b34e58\": rpc error: code = NotFound desc = could not find container \"d2b720a6f46ad342fed6404f83b3f1afb8a32245764be34b4d4a12b964b34e58\": container with ID starting with d2b720a6f46ad342fed6404f83b3f1afb8a32245764be34b4d4a12b964b34e58 not found: ID does not exist"
Jul 20 10:49:14.476903 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.476886    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"]
Jul 20 10:49:14.482198 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.482171    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-pmg5x"]
Jul 20 10:49:14.765386 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:14.765353    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" path="/var/lib/kubelet/pods/e4b6eb1b-51a4-4b44-9a60-b3c3478e914c/volumes"
Jul 20 10:49:15.094027 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.093938    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"]
Jul 20 10:49:15.094221 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.094207    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="storage-initializer"
Jul 20 10:49:15.094221 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.094221    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="storage-initializer"
Jul 20 10:49:15.094314 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.094230    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kube-rbac-proxy"
Jul 20 10:49:15.094314 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.094236    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kube-rbac-proxy"
Jul 20 10:49:15.094314 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.094251    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container"
Jul 20 10:49:15.094314 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.094257    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container"
Jul 20 10:49:15.094314 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.094302    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kserve-container"
Jul 20 10:49:15.094314 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.094310    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b6eb1b-51a4-4b44-9a60-b3c3478e914c" containerName="kube-rbac-proxy"
Jul 20 10:49:15.098690 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.098670    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.101647 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.101629    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:49:15.102480 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.102465    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 20 10:49:15.102525 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.102492    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:49:15.102719 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.102706    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:49:15.102862 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.102850    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 20 10:49:15.103328 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.103316    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:49:15.103376 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.103348    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:49:15.117681 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.117657    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"]
Jul 20 10:49:15.151790 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.151764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/376dc02b-e231-411e-a025-a4369272f280-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.151887 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.151800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/376dc02b-e231-411e-a025-a4369272f280-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.151887 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.151838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/376dc02b-e231-411e-a025-a4369272f280-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.152040 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.151904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/376dc02b-e231-411e-a025-a4369272f280-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.152040 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.151939    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwh6r\" (UniqueName: \"kubernetes.io/projected/376dc02b-e231-411e-a025-a4369272f280-kube-api-access-cwh6r\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.252718 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.252681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/376dc02b-e231-411e-a025-a4369272f280-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.252718 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.252718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/376dc02b-e231-411e-a025-a4369272f280-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.253251 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.252750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/376dc02b-e231-411e-a025-a4369272f280-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.253251 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.252773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/376dc02b-e231-411e-a025-a4369272f280-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.253251 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.252792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwh6r\" (UniqueName: \"kubernetes.io/projected/376dc02b-e231-411e-a025-a4369272f280-kube-api-access-cwh6r\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.253251 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:49:15.252938    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Jul 20 10:49:15.253251 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:49:15.253023    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376dc02b-e231-411e-a025-a4369272f280-proxy-tls podName:376dc02b-e231-411e-a025-a4369272f280 nodeName:}" failed. No retries permitted until 2026-07-20 10:49:15.75300097 +0000 UTC m=+3135.591099547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/376dc02b-e231-411e-a025-a4369272f280-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" (UID: "376dc02b-e231-411e-a025-a4369272f280") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Jul 20 10:49:15.253251 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.253246    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/376dc02b-e231-411e-a025-a4369272f280-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.253469 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.253418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/376dc02b-e231-411e-a025-a4369272f280-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.253469 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.253453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/376dc02b-e231-411e-a025-a4369272f280-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.262557 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.262535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwh6r\" (UniqueName: \"kubernetes.io/projected/376dc02b-e231-411e-a025-a4369272f280-kube-api-access-cwh6r\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.757635 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.757604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/376dc02b-e231-411e-a025-a4369272f280-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:15.759966 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:15.759934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/376dc02b-e231-411e-a025-a4369272f280-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:16.008369 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:16.008288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:16.137599 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:16.137526    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"]
Jul 20 10:49:16.140097 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:49:16.140072    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376dc02b_e231_411e_a025_a4369272f280.slice/crio-0b288c4576c1a115caa2a84ea2acd63857a2b5098f0b1156b31670db29a35347 WatchSource:0}: Error finding container 0b288c4576c1a115caa2a84ea2acd63857a2b5098f0b1156b31670db29a35347: Status 404 returned error can't find the container with id 0b288c4576c1a115caa2a84ea2acd63857a2b5098f0b1156b31670db29a35347
Jul 20 10:49:16.458876 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:16.458839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" event={"ID":"376dc02b-e231-411e-a025-a4369272f280","Type":"ContainerStarted","Data":"332c3423e642982ca863c637d003bfde9528a7830ca36bdc12e5b3e2f197b8e5"}
Jul 20 10:49:16.459262 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:16.458901    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" event={"ID":"376dc02b-e231-411e-a025-a4369272f280","Type":"ContainerStarted","Data":"0b288c4576c1a115caa2a84ea2acd63857a2b5098f0b1156b31670db29a35347"}
Jul 20 10:49:20.479568 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:20.479536    2561 generic.go:356] "Generic (PLEG): container finished" podID="376dc02b-e231-411e-a025-a4369272f280" containerID="332c3423e642982ca863c637d003bfde9528a7830ca36bdc12e5b3e2f197b8e5" exitCode=0
Jul 20 10:49:20.479946 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:20.479608    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" event={"ID":"376dc02b-e231-411e-a025-a4369272f280","Type":"ContainerDied","Data":"332c3423e642982ca863c637d003bfde9528a7830ca36bdc12e5b3e2f197b8e5"}
Jul 20 10:49:21.487507 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:21.487475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" event={"ID":"376dc02b-e231-411e-a025-a4369272f280","Type":"ContainerStarted","Data":"f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde"}
Jul 20 10:49:21.487906 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:21.487533    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" event={"ID":"376dc02b-e231-411e-a025-a4369272f280","Type":"ContainerStarted","Data":"c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252"}
Jul 20 10:49:21.487906 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:21.487808    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:21.488008 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:21.487966    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:21.489350 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:21.489324    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 20 10:49:21.508805 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:21.508763    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podStartSLOduration=2.71899027 podStartE2EDuration="6.508749935s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.789759665s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:49:21.506559971 +0000 UTC m=+3141.344658582" watchObservedRunningTime="2026-07-20 10:49:21.508749935 +0000 UTC m=+3141.346848520"
Jul 20 10:49:22.491405 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:22.491369    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 20 10:49:27.495442 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:27.495412    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:49:27.495934 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:27.495841    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 20 10:49:37.496341 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:37.496297    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 20 10:49:47.495877 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:47.495841    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 20 10:49:57.495968 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:49:57.495932    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 20 10:50:07.496740 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:07.496702    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 20 10:50:17.496052 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:17.496011    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 20 10:50:27.496464 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:27.496427    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 20 10:50:27.761015 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:27.760932    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 20 10:50:37.761277 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:37.761251    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:50:45.215734 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:45.215703    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"]
Jul 20 10:50:45.216223 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:45.215940    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container" containerID="cri-o://c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252" gracePeriod=30
Jul 20 10:50:45.216223 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:45.215997    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kube-rbac-proxy" containerID="cri-o://f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde" gracePeriod=30
Jul 20 10:50:45.778970 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:45.778940    2561 generic.go:356] "Generic (PLEG): container finished" podID="376dc02b-e231-411e-a025-a4369272f280" containerID="f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde" exitCode=2
Jul 20 10:50:45.779168 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:45.779009    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" event={"ID":"376dc02b-e231-411e-a025-a4369272f280","Type":"ContainerDied","Data":"f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde"}
Jul 20 10:50:47.491949 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:47.491909    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Jul 20 10:50:47.761375 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:47.761262    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 20 10:50:49.557603 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.557578    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:50:49.619850 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.619826    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/376dc02b-e231-411e-a025-a4369272f280-cabundle-cert\") pod \"376dc02b-e231-411e-a025-a4369272f280\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") "
Jul 20 10:50:49.620033 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.619875    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/376dc02b-e231-411e-a025-a4369272f280-kube-rbac-proxy-sar-config\") pod \"376dc02b-e231-411e-a025-a4369272f280\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") "
Jul 20 10:50:49.620033 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.619934    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/376dc02b-e231-411e-a025-a4369272f280-proxy-tls\") pod \"376dc02b-e231-411e-a025-a4369272f280\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") "
Jul 20 10:50:49.620033 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.619971    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/376dc02b-e231-411e-a025-a4369272f280-kserve-provision-location\") pod \"376dc02b-e231-411e-a025-a4369272f280\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") "
Jul 20 10:50:49.620033 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.620017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwh6r\" (UniqueName: \"kubernetes.io/projected/376dc02b-e231-411e-a025-a4369272f280-kube-api-access-cwh6r\") pod \"376dc02b-e231-411e-a025-a4369272f280\" (UID: \"376dc02b-e231-411e-a025-a4369272f280\") "
Jul 20 10:50:49.620362 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.620330    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376dc02b-e231-411e-a025-a4369272f280-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "376dc02b-e231-411e-a025-a4369272f280" (UID: "376dc02b-e231-411e-a025-a4369272f280"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:50:49.620362 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.620351    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376dc02b-e231-411e-a025-a4369272f280-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "376dc02b-e231-411e-a025-a4369272f280" (UID: "376dc02b-e231-411e-a025-a4369272f280"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:50:49.620362 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.620354    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376dc02b-e231-411e-a025-a4369272f280-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "376dc02b-e231-411e-a025-a4369272f280" (UID: "376dc02b-e231-411e-a025-a4369272f280"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:50:49.621924 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.621900    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/376dc02b-e231-411e-a025-a4369272f280-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "376dc02b-e231-411e-a025-a4369272f280" (UID: "376dc02b-e231-411e-a025-a4369272f280"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:50:49.622035 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.622012    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376dc02b-e231-411e-a025-a4369272f280-kube-api-access-cwh6r" (OuterVolumeSpecName: "kube-api-access-cwh6r") pod "376dc02b-e231-411e-a025-a4369272f280" (UID: "376dc02b-e231-411e-a025-a4369272f280"). InnerVolumeSpecName "kube-api-access-cwh6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:50:49.720957 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.720937    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwh6r\" (UniqueName: \"kubernetes.io/projected/376dc02b-e231-411e-a025-a4369272f280-kube-api-access-cwh6r\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:50:49.720957 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.720958    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/376dc02b-e231-411e-a025-a4369272f280-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:50:49.721104 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.720968    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/376dc02b-e231-411e-a025-a4369272f280-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:50:49.721104 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.720980    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/376dc02b-e231-411e-a025-a4369272f280-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:50:49.721104 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.720990    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/376dc02b-e231-411e-a025-a4369272f280-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:50:49.794791 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.794761    2561 generic.go:356] "Generic (PLEG): container finished" podID="376dc02b-e231-411e-a025-a4369272f280" containerID="c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252" exitCode=0
Jul 20 10:50:49.794933 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.794802    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" event={"ID":"376dc02b-e231-411e-a025-a4369272f280","Type":"ContainerDied","Data":"c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252"}
Jul 20 10:50:49.794933 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.794821    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"
Jul 20 10:50:49.794933 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.794857    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn" event={"ID":"376dc02b-e231-411e-a025-a4369272f280","Type":"ContainerDied","Data":"0b288c4576c1a115caa2a84ea2acd63857a2b5098f0b1156b31670db29a35347"}
Jul 20 10:50:49.794933 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.794893    2561 scope.go:117] "RemoveContainer" containerID="f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde"
Jul 20 10:50:49.803173 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.803154    2561 scope.go:117] "RemoveContainer" containerID="c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252"
Jul 20 10:50:49.810257 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.810240    2561 scope.go:117] "RemoveContainer" containerID="332c3423e642982ca863c637d003bfde9528a7830ca36bdc12e5b3e2f197b8e5"
Jul 20 10:50:49.817307 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.817288    2561 scope.go:117] "RemoveContainer" containerID="f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde"
Jul 20 10:50:49.817575 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:50:49.817556    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde\": container with ID starting with f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde not found: ID does not exist" containerID="f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde"
Jul 20 10:50:49.817633 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.817584    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde"} err="failed to get container status \"f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde\": rpc error: code = NotFound desc = could not find container \"f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde\": container with ID starting with f7fd6622820e03c5778492c37b677d2b9bf536a21a05d4d05509bca5d3cf3fde not found: ID does not exist"
Jul 20 10:50:49.817633 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.817601    2561 scope.go:117] "RemoveContainer" containerID="c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252"
Jul 20 10:50:49.817853 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:50:49.817837    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252\": container with ID starting with c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252 not found: ID does not exist" containerID="c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252"
Jul 20 10:50:49.817897 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.817860    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252"} err="failed to get container status \"c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252\": rpc error: code = NotFound desc = could not find container \"c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252\": container with ID starting with c503e8f6a7844c45c83d43a0019d83aba8074f76052c0d5ba4b3608800f77252 not found: ID does not exist"
Jul 20 10:50:49.817897 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.817879    2561 scope.go:117] "RemoveContainer" containerID="332c3423e642982ca863c637d003bfde9528a7830ca36bdc12e5b3e2f197b8e5"
Jul 20 10:50:49.818059 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.818042    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"]
Jul 20 10:50:49.818102 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:50:49.818073    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332c3423e642982ca863c637d003bfde9528a7830ca36bdc12e5b3e2f197b8e5\": container with ID starting with 332c3423e642982ca863c637d003bfde9528a7830ca36bdc12e5b3e2f197b8e5 not found: ID does not exist" containerID="332c3423e642982ca863c637d003bfde9528a7830ca36bdc12e5b3e2f197b8e5"
Jul 20 10:50:49.818102 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.818087    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332c3423e642982ca863c637d003bfde9528a7830ca36bdc12e5b3e2f197b8e5"} err="failed to get container status \"332c3423e642982ca863c637d003bfde9528a7830ca36bdc12e5b3e2f197b8e5\": rpc error: code = NotFound desc = could not find container \"332c3423e642982ca863c637d003bfde9528a7830ca36bdc12e5b3e2f197b8e5\": container with ID starting with 332c3423e642982ca863c637d003bfde9528a7830ca36bdc12e5b3e2f197b8e5 not found: ID does not exist"
Jul 20 10:50:49.821839 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:49.821819    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fmxmn"]
Jul 20 10:50:50.353170 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.353135    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"]
Jul 20 10:50:50.353501 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.353475    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kube-rbac-proxy"
Jul 20 10:50:50.353588 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.353503    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kube-rbac-proxy"
Jul 20 10:50:50.353588 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.353528    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="storage-initializer"
Jul 20 10:50:50.353588 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.353537    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="storage-initializer"
Jul 20 10:50:50.353588 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.353561    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container"
Jul 20 10:50:50.353588 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.353570    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container"
Jul 20 10:50:50.353857 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.353637    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kube-rbac-proxy"
Jul 20 10:50:50.353857 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.353648    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="376dc02b-e231-411e-a025-a4369272f280" containerName="kserve-container"
Jul 20 10:50:50.358331 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.358308    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.360819 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.360768    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:50:50.360949 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.360820    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:50:50.360949 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.360801    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 20 10:50:50.361092 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.361064    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:50:50.361263 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.361173    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 20 10:50:50.361875 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.361855    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:50:50.361875 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.361865    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:50:50.366512 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.366492    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"]
Jul 20 10:50:50.424944 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.424910    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96a38532-b13a-4740-845b-d73bbb43343f-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.425163 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.424972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96a38532-b13a-4740-845b-d73bbb43343f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.425163 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.425051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96a38532-b13a-4740-845b-d73bbb43343f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.425163 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.425107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96a38532-b13a-4740-845b-d73bbb43343f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.425163 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.425153    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfwk\" (UniqueName: \"kubernetes.io/projected/96a38532-b13a-4740-845b-d73bbb43343f-kube-api-access-bnfwk\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.526362 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.526331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96a38532-b13a-4740-845b-d73bbb43343f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.526541 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.526377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96a38532-b13a-4740-845b-d73bbb43343f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.526541 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.526506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96a38532-b13a-4740-845b-d73bbb43343f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.526660 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.526545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfwk\" (UniqueName: \"kubernetes.io/projected/96a38532-b13a-4740-845b-d73bbb43343f-kube-api-access-bnfwk\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.526660 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.526573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96a38532-b13a-4740-845b-d73bbb43343f-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.526660 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:50:50.526615    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 20 10:50:50.526796 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:50:50.526671    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a38532-b13a-4740-845b-d73bbb43343f-proxy-tls podName:96a38532-b13a-4740-845b-d73bbb43343f nodeName:}" failed. No retries permitted until 2026-07-20 10:50:51.026655104 +0000 UTC m=+3230.864753666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/96a38532-b13a-4740-845b-d73bbb43343f-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" (UID: "96a38532-b13a-4740-845b-d73bbb43343f") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 20 10:50:50.526796 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.526786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96a38532-b13a-4740-845b-d73bbb43343f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.527069 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.527048    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96a38532-b13a-4740-845b-d73bbb43343f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.527288 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.527270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96a38532-b13a-4740-845b-d73bbb43343f-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.535002 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.534982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfwk\" (UniqueName: \"kubernetes.io/projected/96a38532-b13a-4740-845b-d73bbb43343f-kube-api-access-bnfwk\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:50.765232 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:50.765201    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="376dc02b-e231-411e-a025-a4369272f280" path="/var/lib/kubelet/pods/376dc02b-e231-411e-a025-a4369272f280/volumes"
Jul 20 10:50:51.030870 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:51.030781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96a38532-b13a-4740-845b-d73bbb43343f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:51.033169 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:51.033150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96a38532-b13a-4740-845b-d73bbb43343f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:51.268998 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:51.268948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:51.397597 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:51.397573    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"]
Jul 20 10:50:51.399665 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:50:51.399639    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a38532_b13a_4740_845b_d73bbb43343f.slice/crio-76a406979079bd02f86febe4dbdff56403086534cba03a0134d453745c4bda0b WatchSource:0}: Error finding container 76a406979079bd02f86febe4dbdff56403086534cba03a0134d453745c4bda0b: Status 404 returned error can't find the container with id 76a406979079bd02f86febe4dbdff56403086534cba03a0134d453745c4bda0b
Jul 20 10:50:51.804226 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:51.804187    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" event={"ID":"96a38532-b13a-4740-845b-d73bbb43343f","Type":"ContainerStarted","Data":"4ef6f03b1f9308c291a331bbcb43bed517e3c4f9071a0bd7c7e31fca06a4a9cd"}
Jul 20 10:50:51.804572 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:51.804254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" event={"ID":"96a38532-b13a-4740-845b-d73bbb43343f","Type":"ContainerStarted","Data":"76a406979079bd02f86febe4dbdff56403086534cba03a0134d453745c4bda0b"}
Jul 20 10:50:55.820499 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:55.820469    2561 generic.go:356] "Generic (PLEG): container finished" podID="96a38532-b13a-4740-845b-d73bbb43343f" containerID="4ef6f03b1f9308c291a331bbcb43bed517e3c4f9071a0bd7c7e31fca06a4a9cd" exitCode=0
Jul 20 10:50:55.820879 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:55.820540    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" event={"ID":"96a38532-b13a-4740-845b-d73bbb43343f","Type":"ContainerDied","Data":"4ef6f03b1f9308c291a331bbcb43bed517e3c4f9071a0bd7c7e31fca06a4a9cd"}
Jul 20 10:50:56.826241 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:56.826206    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" event={"ID":"96a38532-b13a-4740-845b-d73bbb43343f","Type":"ContainerStarted","Data":"da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066"}
Jul 20 10:50:56.826632 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:56.826266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" event={"ID":"96a38532-b13a-4740-845b-d73bbb43343f","Type":"ContainerStarted","Data":"1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46"}
Jul 20 10:50:56.826632 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:56.826467    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:56.826632 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:56.826603    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:50:56.848687 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:50:56.848640    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" podStartSLOduration=3.228455257 podStartE2EDuration="6.84862824s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.620172983s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:50:56.846542595 +0000 UTC m=+3236.684641200" watchObservedRunningTime="2026-07-20 10:50:56.84862824 +0000 UTC m=+3236.686726824"
Jul 20 10:51:02.835332 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:51:02.835304    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:51:32.836234 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:51:32.836197    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 20 10:51:42.836456 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:51:42.836420    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 20 10:51:52.836052 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:51:52.836013    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 20 10:52:02.836749 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:02.836711    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 20 10:52:13.763451 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:13.763402    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-predictive-sklearn-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 10:52:16.164219 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:16.164190    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:52:16.169582 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:16.169562    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:52:22.764690 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:22.764613    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:52:30.502541 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:30.502505    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"]
Jul 20 10:52:30.502932 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:30.502758    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kserve-container" containerID="cri-o://1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46" gracePeriod=30
Jul 20 10:52:30.502932 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:30.502800    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kube-rbac-proxy" containerID="cri-o://da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066" gracePeriod=30
Jul 20 10:52:31.167966 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:31.167934    2561 generic.go:356] "Generic (PLEG): container finished" podID="96a38532-b13a-4740-845b-d73bbb43343f" containerID="da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066" exitCode=2
Jul 20 10:52:31.168151 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:31.167994    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" event={"ID":"96a38532-b13a-4740-845b-d73bbb43343f","Type":"ContainerDied","Data":"da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066"}
Jul 20 10:52:32.762068 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:32.762024    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 20 10:52:32.831201 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:32.831167    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Jul 20 10:52:34.743625 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.743600    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:52:34.859162 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.859063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96a38532-b13a-4740-845b-d73bbb43343f-kserve-provision-location\") pod \"96a38532-b13a-4740-845b-d73bbb43343f\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") "
Jul 20 10:52:34.859162 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.859149    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96a38532-b13a-4740-845b-d73bbb43343f-cabundle-cert\") pod \"96a38532-b13a-4740-845b-d73bbb43343f\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") "
Jul 20 10:52:34.859162 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.859171    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96a38532-b13a-4740-845b-d73bbb43343f-kube-rbac-proxy-sar-config\") pod \"96a38532-b13a-4740-845b-d73bbb43343f\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") "
Jul 20 10:52:34.859443 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.859202    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96a38532-b13a-4740-845b-d73bbb43343f-proxy-tls\") pod \"96a38532-b13a-4740-845b-d73bbb43343f\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") "
Jul 20 10:52:34.859443 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.859222    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnfwk\" (UniqueName: \"kubernetes.io/projected/96a38532-b13a-4740-845b-d73bbb43343f-kube-api-access-bnfwk\") pod \"96a38532-b13a-4740-845b-d73bbb43343f\" (UID: \"96a38532-b13a-4740-845b-d73bbb43343f\") "
Jul 20 10:52:34.859443 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.859391    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a38532-b13a-4740-845b-d73bbb43343f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96a38532-b13a-4740-845b-d73bbb43343f" (UID: "96a38532-b13a-4740-845b-d73bbb43343f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:52:34.859605 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.859516    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a38532-b13a-4740-845b-d73bbb43343f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "96a38532-b13a-4740-845b-d73bbb43343f" (UID: "96a38532-b13a-4740-845b-d73bbb43343f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:52:34.859605 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.859553    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a38532-b13a-4740-845b-d73bbb43343f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96a38532-b13a-4740-845b-d73bbb43343f" (UID: "96a38532-b13a-4740-845b-d73bbb43343f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:52:34.861331 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.861302    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a38532-b13a-4740-845b-d73bbb43343f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96a38532-b13a-4740-845b-d73bbb43343f" (UID: "96a38532-b13a-4740-845b-d73bbb43343f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:52:34.861436 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.861406    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a38532-b13a-4740-845b-d73bbb43343f-kube-api-access-bnfwk" (OuterVolumeSpecName: "kube-api-access-bnfwk") pod "96a38532-b13a-4740-845b-d73bbb43343f" (UID: "96a38532-b13a-4740-845b-d73bbb43343f"). InnerVolumeSpecName "kube-api-access-bnfwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:52:34.959836 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.959812    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96a38532-b13a-4740-845b-d73bbb43343f-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:52:34.959836 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.959834    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96a38532-b13a-4740-845b-d73bbb43343f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:52:34.959981 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.959845    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96a38532-b13a-4740-845b-d73bbb43343f-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:52:34.959981 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.959855    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnfwk\" (UniqueName: \"kubernetes.io/projected/96a38532-b13a-4740-845b-d73bbb43343f-kube-api-access-bnfwk\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:52:34.959981 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:34.959863    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96a38532-b13a-4740-845b-d73bbb43343f-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:52:35.184123 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.184082    2561 generic.go:356] "Generic (PLEG): container finished" podID="96a38532-b13a-4740-845b-d73bbb43343f" containerID="1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46" exitCode=0
Jul 20 10:52:35.184273 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.184156    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"
Jul 20 10:52:35.184273 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.184153    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" event={"ID":"96a38532-b13a-4740-845b-d73bbb43343f","Type":"ContainerDied","Data":"1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46"}
Jul 20 10:52:35.184355 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.184301    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2" event={"ID":"96a38532-b13a-4740-845b-d73bbb43343f","Type":"ContainerDied","Data":"76a406979079bd02f86febe4dbdff56403086534cba03a0134d453745c4bda0b"}
Jul 20 10:52:35.184355 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.184329    2561 scope.go:117] "RemoveContainer" containerID="da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066"
Jul 20 10:52:35.192978 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.192958    2561 scope.go:117] "RemoveContainer" containerID="1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46"
Jul 20 10:52:35.199759 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.199741    2561 scope.go:117] "RemoveContainer" containerID="4ef6f03b1f9308c291a331bbcb43bed517e3c4f9071a0bd7c7e31fca06a4a9cd"
Jul 20 10:52:35.206123 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.206090    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"]
Jul 20 10:52:35.206662 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.206646    2561 scope.go:117] "RemoveContainer" containerID="da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066"
Jul 20 10:52:35.206905 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:52:35.206888    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066\": container with ID starting with da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066 not found: ID does not exist" containerID="da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066"
Jul 20 10:52:35.206955 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.206918    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066"} err="failed to get container status \"da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066\": rpc error: code = NotFound desc = could not find container \"da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066\": container with ID starting with da08307362b3c67a73ab3c3b6d2c93a6734bc43e617a561da85922b28a38b066 not found: ID does not exist"
Jul 20 10:52:35.206955 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.206934    2561 scope.go:117] "RemoveContainer" containerID="1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46"
Jul 20 10:52:35.207194 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:52:35.207173    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46\": container with ID starting with 1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46 not found: ID does not exist" containerID="1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46"
Jul 20 10:52:35.207260 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.207200    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46"} err="failed to get container status \"1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46\": rpc error: code = NotFound desc = could not find container \"1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46\": container with ID starting with 1f4b4dce88c3517879737ba476c62729d070f804873ab08d6f8b4f5eb6b1fc46 not found: ID does not exist"
Jul 20 10:52:35.207260 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.207218    2561 scope.go:117] "RemoveContainer" containerID="4ef6f03b1f9308c291a331bbcb43bed517e3c4f9071a0bd7c7e31fca06a4a9cd"
Jul 20 10:52:35.207532 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:52:35.207475    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef6f03b1f9308c291a331bbcb43bed517e3c4f9071a0bd7c7e31fca06a4a9cd\": container with ID starting with 4ef6f03b1f9308c291a331bbcb43bed517e3c4f9071a0bd7c7e31fca06a4a9cd not found: ID does not exist" containerID="4ef6f03b1f9308c291a331bbcb43bed517e3c4f9071a0bd7c7e31fca06a4a9cd"
Jul 20 10:52:35.207532 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.207507    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef6f03b1f9308c291a331bbcb43bed517e3c4f9071a0bd7c7e31fca06a4a9cd"} err="failed to get container status \"4ef6f03b1f9308c291a331bbcb43bed517e3c4f9071a0bd7c7e31fca06a4a9cd\": rpc error: code = NotFound desc = could not find container \"4ef6f03b1f9308c291a331bbcb43bed517e3c4f9071a0bd7c7e31fca06a4a9cd\": container with ID starting with 4ef6f03b1f9308c291a331bbcb43bed517e3c4f9071a0bd7c7e31fca06a4a9cd not found: ID does not exist"
Jul 20 10:52:35.209829 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.209807    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-jb7j2"]
Jul 20 10:52:35.667428 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.667394    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"]
Jul 20 10:52:35.667655 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.667641    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kube-rbac-proxy"
Jul 20 10:52:35.667700 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.667657    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kube-rbac-proxy"
Jul 20 10:52:35.667700 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.667668    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="storage-initializer"
Jul 20 10:52:35.667700 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.667674    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="storage-initializer"
Jul 20 10:52:35.667700 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.667686    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kserve-container"
Jul 20 10:52:35.667700 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.667692    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kserve-container"
Jul 20 10:52:35.667852 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.667735    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kserve-container"
Jul 20 10:52:35.667852 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.667745    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a38532-b13a-4740-845b-d73bbb43343f" containerName="kube-rbac-proxy"
Jul 20 10:52:35.672022 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.672001    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.674670 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.674626    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:52:35.674850 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.674807    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:52:35.674957 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.674895    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 20 10:52:35.675075 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.675062    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:52:35.675745 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.675727    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:52:35.675834 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.675732    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 20 10:52:35.675834 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.675769    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:52:35.681071 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.681020    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"]
Jul 20 10:52:35.764220 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.764184    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpp54\" (UniqueName: \"kubernetes.io/projected/8cbab199-65ba-48be-8769-48f3e4bf0a55-kube-api-access-vpp54\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.764579 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.764229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cbab199-65ba-48be-8769-48f3e4bf0a55-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.764579 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.764294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cbab199-65ba-48be-8769-48f3e4bf0a55-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.764579 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.764348    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cbab199-65ba-48be-8769-48f3e4bf0a55-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.764579 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.764369    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cbab199-65ba-48be-8769-48f3e4bf0a55-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.864960 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.864931    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cbab199-65ba-48be-8769-48f3e4bf0a55-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.865163 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.864979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cbab199-65ba-48be-8769-48f3e4bf0a55-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.865163 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.865021    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cbab199-65ba-48be-8769-48f3e4bf0a55-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.865163 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.865051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cbab199-65ba-48be-8769-48f3e4bf0a55-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.865163 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.865098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpp54\" (UniqueName: \"kubernetes.io/projected/8cbab199-65ba-48be-8769-48f3e4bf0a55-kube-api-access-vpp54\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.865496 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.865464    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cbab199-65ba-48be-8769-48f3e4bf0a55-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.865618 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.865599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cbab199-65ba-48be-8769-48f3e4bf0a55-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.865700 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.865683    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cbab199-65ba-48be-8769-48f3e4bf0a55-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.867579 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.867562    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cbab199-65ba-48be-8769-48f3e4bf0a55-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.872587 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.872566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpp54\" (UniqueName: \"kubernetes.io/projected/8cbab199-65ba-48be-8769-48f3e4bf0a55-kube-api-access-vpp54\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:35.983154 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:35.983071    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:36.109328 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:36.109294    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"]
Jul 20 10:52:36.112838 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:52:36.112809    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbab199_65ba_48be_8769_48f3e4bf0a55.slice/crio-61d8fca39ab546b9649df45b2d2e29c4c2e54a811e36b52abcc2af411ce43e29 WatchSource:0}: Error finding container 61d8fca39ab546b9649df45b2d2e29c4c2e54a811e36b52abcc2af411ce43e29: Status 404 returned error can't find the container with id 61d8fca39ab546b9649df45b2d2e29c4c2e54a811e36b52abcc2af411ce43e29
Jul 20 10:52:36.191018 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:36.190989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" event={"ID":"8cbab199-65ba-48be-8769-48f3e4bf0a55","Type":"ContainerStarted","Data":"fd4ec4ab80decd1d60c6a73da333d3e5012928a09c09d32b2643924f66868640"}
Jul 20 10:52:36.191195 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:36.191042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" event={"ID":"8cbab199-65ba-48be-8769-48f3e4bf0a55","Type":"ContainerStarted","Data":"61d8fca39ab546b9649df45b2d2e29c4c2e54a811e36b52abcc2af411ce43e29"}
Jul 20 10:52:36.765045 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:36.765011    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a38532-b13a-4740-845b-d73bbb43343f" path="/var/lib/kubelet/pods/96a38532-b13a-4740-845b-d73bbb43343f/volumes"
Jul 20 10:52:40.207372 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:40.207341    2561 generic.go:356] "Generic (PLEG): container finished" podID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerID="fd4ec4ab80decd1d60c6a73da333d3e5012928a09c09d32b2643924f66868640" exitCode=0
Jul 20 10:52:40.207743 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:40.207401    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" event={"ID":"8cbab199-65ba-48be-8769-48f3e4bf0a55","Type":"ContainerDied","Data":"fd4ec4ab80decd1d60c6a73da333d3e5012928a09c09d32b2643924f66868640"}
Jul 20 10:52:41.214441 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:41.214409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" event={"ID":"8cbab199-65ba-48be-8769-48f3e4bf0a55","Type":"ContainerStarted","Data":"558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0"}
Jul 20 10:52:41.214962 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:41.214479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" event={"ID":"8cbab199-65ba-48be-8769-48f3e4bf0a55","Type":"ContainerStarted","Data":"5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f"}
Jul 20 10:52:41.214962 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:41.214704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:41.237885 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:41.237843    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" podStartSLOduration=2.434701681 podStartE2EDuration="6.237830216s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.803128535s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:52:41.235960919 +0000 UTC m=+3341.074059522" watchObservedRunningTime="2026-07-20 10:52:41.237830216 +0000 UTC m=+3341.075928800"
Jul 20 10:52:42.219086 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:42.219054    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:52:48.228183 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:52:48.228154    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:53:18.228936 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:53:18.228892    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.52:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 20 10:53:28.228615 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:53:28.228573    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.52:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 20 10:53:38.229064 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:53:38.229026    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.52:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 20 10:53:48.228591 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:53:48.228552    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.52:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 20 10:53:58.261266 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:53:58.261190    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:54:05.786446 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:05.786412    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"]
Jul 20 10:54:05.787041 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:05.786696    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kserve-container" containerID="cri-o://5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f" gracePeriod=30
Jul 20 10:54:05.787041 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:05.786767    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kube-rbac-proxy" containerID="cri-o://558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0" gracePeriod=30
Jul 20 10:54:06.524048 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:06.524017    2561 generic.go:356] "Generic (PLEG): container finished" podID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerID="558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0" exitCode=2
Jul 20 10:54:06.524231 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:06.524088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" event={"ID":"8cbab199-65ba-48be-8769-48f3e4bf0a55","Type":"ContainerDied","Data":"558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0"}
Jul 20 10:54:08.223266 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:08.223221    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 20 10:54:08.229507 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:08.229471    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.52:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 20 10:54:09.832525 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:09.832502    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:54:09.919860 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:09.919796    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cbab199-65ba-48be-8769-48f3e4bf0a55-proxy-tls\") pod \"8cbab199-65ba-48be-8769-48f3e4bf0a55\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") "
Jul 20 10:54:09.919860 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:09.919837    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cbab199-65ba-48be-8769-48f3e4bf0a55-kube-rbac-proxy-sar-config\") pod \"8cbab199-65ba-48be-8769-48f3e4bf0a55\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") "
Jul 20 10:54:09.920026 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:09.919935    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpp54\" (UniqueName: \"kubernetes.io/projected/8cbab199-65ba-48be-8769-48f3e4bf0a55-kube-api-access-vpp54\") pod \"8cbab199-65ba-48be-8769-48f3e4bf0a55\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") "
Jul 20 10:54:09.920026 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:09.919979    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cbab199-65ba-48be-8769-48f3e4bf0a55-cabundle-cert\") pod \"8cbab199-65ba-48be-8769-48f3e4bf0a55\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") "
Jul 20 10:54:09.920109 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:09.920045    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cbab199-65ba-48be-8769-48f3e4bf0a55-kserve-provision-location\") pod \"8cbab199-65ba-48be-8769-48f3e4bf0a55\" (UID: \"8cbab199-65ba-48be-8769-48f3e4bf0a55\") "
Jul 20 10:54:09.920200 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:09.920150    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbab199-65ba-48be-8769-48f3e4bf0a55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8cbab199-65ba-48be-8769-48f3e4bf0a55" (UID: "8cbab199-65ba-48be-8769-48f3e4bf0a55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:54:09.920259 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:09.920237    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cbab199-65ba-48be-8769-48f3e4bf0a55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:54:09.920363 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:09.920341    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbab199-65ba-48be-8769-48f3e4bf0a55-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8cbab199-65ba-48be-8769-48f3e4bf0a55" (UID: "8cbab199-65ba-48be-8769-48f3e4bf0a55"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:54:09.920421 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:09.920379    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbab199-65ba-48be-8769-48f3e4bf0a55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8cbab199-65ba-48be-8769-48f3e4bf0a55" (UID: "8cbab199-65ba-48be-8769-48f3e4bf0a55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:54:09.921980 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:09.921962    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbab199-65ba-48be-8769-48f3e4bf0a55-kube-api-access-vpp54" (OuterVolumeSpecName: "kube-api-access-vpp54") pod "8cbab199-65ba-48be-8769-48f3e4bf0a55" (UID: "8cbab199-65ba-48be-8769-48f3e4bf0a55"). InnerVolumeSpecName "kube-api-access-vpp54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:54:09.922047 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:09.921976    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbab199-65ba-48be-8769-48f3e4bf0a55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8cbab199-65ba-48be-8769-48f3e4bf0a55" (UID: "8cbab199-65ba-48be-8769-48f3e4bf0a55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:54:10.021142 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.021093    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cbab199-65ba-48be-8769-48f3e4bf0a55-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:54:10.021142 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.021142    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cbab199-65ba-48be-8769-48f3e4bf0a55-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:54:10.021293 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.021157    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cbab199-65ba-48be-8769-48f3e4bf0a55-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:54:10.021293 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.021171    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpp54\" (UniqueName: \"kubernetes.io/projected/8cbab199-65ba-48be-8769-48f3e4bf0a55-kube-api-access-vpp54\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:54:10.542782 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.542750    2561 generic.go:356] "Generic (PLEG): container finished" podID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerID="5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f" exitCode=0
Jul 20 10:54:10.542965 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.542810    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" event={"ID":"8cbab199-65ba-48be-8769-48f3e4bf0a55","Type":"ContainerDied","Data":"5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f"}
Jul 20 10:54:10.542965 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.542871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8" event={"ID":"8cbab199-65ba-48be-8769-48f3e4bf0a55","Type":"ContainerDied","Data":"61d8fca39ab546b9649df45b2d2e29c4c2e54a811e36b52abcc2af411ce43e29"}
Jul 20 10:54:10.542965 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.542890    2561 scope.go:117] "RemoveContainer" containerID="558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0"
Jul 20 10:54:10.542965 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.542876    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"
Jul 20 10:54:10.551286 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.551266    2561 scope.go:117] "RemoveContainer" containerID="5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f"
Jul 20 10:54:10.558460 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.558443    2561 scope.go:117] "RemoveContainer" containerID="fd4ec4ab80decd1d60c6a73da333d3e5012928a09c09d32b2643924f66868640"
Jul 20 10:54:10.563606 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.563581    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"]
Jul 20 10:54:10.566005 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.565921    2561 scope.go:117] "RemoveContainer" containerID="558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0"
Jul 20 10:54:10.566264 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:54:10.566236    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0\": container with ID starting with 558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0 not found: ID does not exist" containerID="558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0"
Jul 20 10:54:10.566362 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.566275    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0"} err="failed to get container status \"558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0\": rpc error: code = NotFound desc = could not find container \"558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0\": container with ID starting with 558a1c7fa9ecddf72191ed6c4429736c8f3bf726d05905e747ccc0aa05d7c6a0 not found: ID does not exist"
Jul 20 10:54:10.566362 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.566299    2561 scope.go:117] "RemoveContainer" containerID="5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f"
Jul 20 10:54:10.566561 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:54:10.566541    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f\": container with ID starting with 5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f not found: ID does not exist" containerID="5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f"
Jul 20 10:54:10.566631 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.566572    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f"} err="failed to get container status \"5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f\": rpc error: code = NotFound desc = could not find container \"5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f\": container with ID starting with 5c3bd8c3ec431048754e15dfbecb5435609188ad6ab608943ec5094773d58e9f not found: ID does not exist"
Jul 20 10:54:10.566631 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.566594    2561 scope.go:117] "RemoveContainer" containerID="fd4ec4ab80decd1d60c6a73da333d3e5012928a09c09d32b2643924f66868640"
Jul 20 10:54:10.566853 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:54:10.566835    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4ec4ab80decd1d60c6a73da333d3e5012928a09c09d32b2643924f66868640\": container with ID starting with fd4ec4ab80decd1d60c6a73da333d3e5012928a09c09d32b2643924f66868640 not found: ID does not exist" containerID="fd4ec4ab80decd1d60c6a73da333d3e5012928a09c09d32b2643924f66868640"
Jul 20 10:54:10.566853 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.566858    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4ec4ab80decd1d60c6a73da333d3e5012928a09c09d32b2643924f66868640"} err="failed to get container status \"fd4ec4ab80decd1d60c6a73da333d3e5012928a09c09d32b2643924f66868640\": rpc error: code = NotFound desc = could not find container \"fd4ec4ab80decd1d60c6a73da333d3e5012928a09c09d32b2643924f66868640\": container with ID starting with fd4ec4ab80decd1d60c6a73da333d3e5012928a09c09d32b2643924f66868640 not found: ID does not exist"
Jul 20 10:54:10.567323 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.567308    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-tw9q8"]
Jul 20 10:54:10.765784 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.765752    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" path="/var/lib/kubelet/pods/8cbab199-65ba-48be-8769-48f3e4bf0a55/volumes"
Jul 20 10:54:10.948940 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.948909    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"]
Jul 20 10:54:10.949439 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.949274    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kube-rbac-proxy"
Jul 20 10:54:10.949439 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.949292    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kube-rbac-proxy"
Jul 20 10:54:10.949439 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.949313    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="storage-initializer"
Jul 20 10:54:10.949439 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.949321    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="storage-initializer"
Jul 20 10:54:10.949439 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.949332    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kserve-container"
Jul 20 10:54:10.949439 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.949340    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kserve-container"
Jul 20 10:54:10.949439 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.949419    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kserve-container"
Jul 20 10:54:10.949439 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.949432    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cbab199-65ba-48be-8769-48f3e4bf0a55" containerName="kube-rbac-proxy"
Jul 20 10:54:10.954199 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.954178    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:10.956934 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.956912    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:54:10.957032 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.956912    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:54:10.957032 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.956912    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:54:10.957175 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.957156    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 20 10:54:10.957256 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.957240    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:54:10.957319 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.957237    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 20 10:54:10.957319 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.957243    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:54:10.962322 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:10.962302    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"]
Jul 20 10:54:11.026547 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.026518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c982-dacb-4683-a183-536f41908bff-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.026547 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.026548    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5f1c982-dacb-4683-a183-536f41908bff-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.026710 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.026576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8zk\" (UniqueName: \"kubernetes.io/projected/b5f1c982-dacb-4683-a183-536f41908bff-kube-api-access-fm8zk\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.026710 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.026598    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5f1c982-dacb-4683-a183-536f41908bff-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.026710 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.026636    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5f1c982-dacb-4683-a183-536f41908bff-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.127477 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.127452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5f1c982-dacb-4683-a183-536f41908bff-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.127581 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.127482    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c982-dacb-4683-a183-536f41908bff-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.127581 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.127508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5f1c982-dacb-4683-a183-536f41908bff-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.127581 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.127549    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8zk\" (UniqueName: \"kubernetes.io/projected/b5f1c982-dacb-4683-a183-536f41908bff-kube-api-access-fm8zk\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.127581 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.127576    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5f1c982-dacb-4683-a183-536f41908bff-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.127765 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:54:11.127597    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Jul 20 10:54:11.127765 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:54:11.127667    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f1c982-dacb-4683-a183-536f41908bff-proxy-tls podName:b5f1c982-dacb-4683-a183-536f41908bff nodeName:}" failed. No retries permitted until 2026-07-20 10:54:11.627647918 +0000 UTC m=+3431.465746482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5f1c982-dacb-4683-a183-536f41908bff-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" (UID: "b5f1c982-dacb-4683-a183-536f41908bff") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Jul 20 10:54:11.127908 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.127893    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c982-dacb-4683-a183-536f41908bff-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.128239 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.128224    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5f1c982-dacb-4683-a183-536f41908bff-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.128239 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.128231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5f1c982-dacb-4683-a183-536f41908bff-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.136401 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.136384    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8zk\" (UniqueName: \"kubernetes.io/projected/b5f1c982-dacb-4683-a183-536f41908bff-kube-api-access-fm8zk\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.632129 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.632074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5f1c982-dacb-4683-a183-536f41908bff-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.634517 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.634497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5f1c982-dacb-4683-a183-536f41908bff-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.864514 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.864475    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:11.988482 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:11.988447    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"]
Jul 20 10:54:11.991790 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:54:11.991762    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f1c982_dacb_4683_a183_536f41908bff.slice/crio-a72119e6cecdceed3749d180533fbf6de3f5de1e714a61be7adfc1c3d0f2416f WatchSource:0}: Error finding container a72119e6cecdceed3749d180533fbf6de3f5de1e714a61be7adfc1c3d0f2416f: Status 404 returned error can't find the container with id a72119e6cecdceed3749d180533fbf6de3f5de1e714a61be7adfc1c3d0f2416f
Jul 20 10:54:12.552208 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:12.552167    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" event={"ID":"b5f1c982-dacb-4683-a183-536f41908bff","Type":"ContainerStarted","Data":"8f00f79f9f39a5b132a5b1d3955d29f551e686acd4ebb2f2365f80b48b39df05"}
Jul 20 10:54:12.552386 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:12.552244    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" event={"ID":"b5f1c982-dacb-4683-a183-536f41908bff","Type":"ContainerStarted","Data":"a72119e6cecdceed3749d180533fbf6de3f5de1e714a61be7adfc1c3d0f2416f"}
Jul 20 10:54:16.566836 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:16.566803    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5f1c982-dacb-4683-a183-536f41908bff" containerID="8f00f79f9f39a5b132a5b1d3955d29f551e686acd4ebb2f2365f80b48b39df05" exitCode=0
Jul 20 10:54:16.567248 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:16.566855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" event={"ID":"b5f1c982-dacb-4683-a183-536f41908bff","Type":"ContainerDied","Data":"8f00f79f9f39a5b132a5b1d3955d29f551e686acd4ebb2f2365f80b48b39df05"}
Jul 20 10:54:17.572414 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:17.572384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" event={"ID":"b5f1c982-dacb-4683-a183-536f41908bff","Type":"ContainerStarted","Data":"6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56"}
Jul 20 10:54:17.572773 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:17.572440    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" event={"ID":"b5f1c982-dacb-4683-a183-536f41908bff","Type":"ContainerStarted","Data":"339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df"}
Jul 20 10:54:17.572773 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:17.572622    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:17.608237 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:17.608189    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" podStartSLOduration=3.715993875 podStartE2EDuration="7.608175402s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.892181527s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:54:17.607030135 +0000 UTC m=+3437.445128719" watchObservedRunningTime="2026-07-20 10:54:17.608175402 +0000 UTC m=+3437.446273987"
Jul 20 10:54:18.576361 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:18.576329    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:24.585302 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:24.585274    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:54:54.586587 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:54:54.586546    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 20 10:55:04.586679 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:04.586640    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 20 10:55:14.586652 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:14.586609    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 20 10:55:24.586052 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:24.585974    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 20 10:55:31.861167 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:31.861105    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:55:41.059618 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:41.059585    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"]
Jul 20 10:55:41.060005 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:41.059839    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kube-rbac-proxy" containerID="cri-o://6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56" gracePeriod=30
Jul 20 10:55:41.060005 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:41.059861    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kserve-container" containerID="cri-o://339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df" gracePeriod=30
Jul 20 10:55:41.762238 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:41.762198    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 20 10:55:41.877465 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:41.877435    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5f1c982-dacb-4683-a183-536f41908bff" containerID="6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56" exitCode=2
Jul 20 10:55:41.877465 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:41.877469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" event={"ID":"b5f1c982-dacb-4683-a183-536f41908bff","Type":"ContainerDied","Data":"6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56"}
Jul 20 10:55:44.580756 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:44.580716    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Jul 20 10:55:45.597294 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.597270    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:55:45.691712 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.691686    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5f1c982-dacb-4683-a183-536f41908bff-proxy-tls\") pod \"b5f1c982-dacb-4683-a183-536f41908bff\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") "
Jul 20 10:55:45.691899 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.691728    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm8zk\" (UniqueName: \"kubernetes.io/projected/b5f1c982-dacb-4683-a183-536f41908bff-kube-api-access-fm8zk\") pod \"b5f1c982-dacb-4683-a183-536f41908bff\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") "
Jul 20 10:55:45.691899 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.691756    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5f1c982-dacb-4683-a183-536f41908bff-kube-rbac-proxy-sar-config\") pod \"b5f1c982-dacb-4683-a183-536f41908bff\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") "
Jul 20 10:55:45.691899 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.691780    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c982-dacb-4683-a183-536f41908bff-kserve-provision-location\") pod \"b5f1c982-dacb-4683-a183-536f41908bff\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") "
Jul 20 10:55:45.691899 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.691828    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5f1c982-dacb-4683-a183-536f41908bff-cabundle-cert\") pod \"b5f1c982-dacb-4683-a183-536f41908bff\" (UID: \"b5f1c982-dacb-4683-a183-536f41908bff\") "
Jul 20 10:55:45.692158 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.692125    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f1c982-dacb-4683-a183-536f41908bff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5f1c982-dacb-4683-a183-536f41908bff" (UID: "b5f1c982-dacb-4683-a183-536f41908bff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:55:45.692520 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.692139    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f1c982-dacb-4683-a183-536f41908bff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5f1c982-dacb-4683-a183-536f41908bff" (UID: "b5f1c982-dacb-4683-a183-536f41908bff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:55:45.692520 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.692294    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f1c982-dacb-4683-a183-536f41908bff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5f1c982-dacb-4683-a183-536f41908bff" (UID: "b5f1c982-dacb-4683-a183-536f41908bff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:55:45.693905 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.693882    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f1c982-dacb-4683-a183-536f41908bff-kube-api-access-fm8zk" (OuterVolumeSpecName: "kube-api-access-fm8zk") pod "b5f1c982-dacb-4683-a183-536f41908bff" (UID: "b5f1c982-dacb-4683-a183-536f41908bff"). InnerVolumeSpecName "kube-api-access-fm8zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:55:45.694002 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.693963    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f1c982-dacb-4683-a183-536f41908bff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5f1c982-dacb-4683-a183-536f41908bff" (UID: "b5f1c982-dacb-4683-a183-536f41908bff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:55:45.793267 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.793216    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5f1c982-dacb-4683-a183-536f41908bff-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:55:45.793267 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.793235    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5f1c982-dacb-4683-a183-536f41908bff-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:55:45.793267 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.793243    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm8zk\" (UniqueName: \"kubernetes.io/projected/b5f1c982-dacb-4683-a183-536f41908bff-kube-api-access-fm8zk\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:55:45.793267 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.793252    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5f1c982-dacb-4683-a183-536f41908bff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:55:45.793267 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.793260    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c982-dacb-4683-a183-536f41908bff-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:55:45.897054 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.897019    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5f1c982-dacb-4683-a183-536f41908bff" containerID="339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df" exitCode=0
Jul 20 10:55:45.897231 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.897087    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"
Jul 20 10:55:45.897231 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.897100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" event={"ID":"b5f1c982-dacb-4683-a183-536f41908bff","Type":"ContainerDied","Data":"339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df"}
Jul 20 10:55:45.897231 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.897195    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb" event={"ID":"b5f1c982-dacb-4683-a183-536f41908bff","Type":"ContainerDied","Data":"a72119e6cecdceed3749d180533fbf6de3f5de1e714a61be7adfc1c3d0f2416f"}
Jul 20 10:55:45.897231 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.897223    2561 scope.go:117] "RemoveContainer" containerID="6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56"
Jul 20 10:55:45.905455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.905438    2561 scope.go:117] "RemoveContainer" containerID="339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df"
Jul 20 10:55:45.912435 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.912415    2561 scope.go:117] "RemoveContainer" containerID="8f00f79f9f39a5b132a5b1d3955d29f551e686acd4ebb2f2365f80b48b39df05"
Jul 20 10:55:45.919991 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.919971    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"]
Jul 20 10:55:45.920053 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.920041    2561 scope.go:117] "RemoveContainer" containerID="6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56"
Jul 20 10:55:45.920326 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:55:45.920305    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56\": container with ID starting with 6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56 not found: ID does not exist" containerID="6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56"
Jul 20 10:55:45.920418 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.920336    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56"} err="failed to get container status \"6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56\": rpc error: code = NotFound desc = could not find container \"6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56\": container with ID starting with 6d8365646c865eefdd54a64f8fed377e33eb7dbe8794b5ab640ab477ce34ae56 not found: ID does not exist"
Jul 20 10:55:45.920418 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.920359    2561 scope.go:117] "RemoveContainer" containerID="339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df"
Jul 20 10:55:45.920609 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:55:45.920591    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df\": container with ID starting with 339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df not found: ID does not exist" containerID="339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df"
Jul 20 10:55:45.920655 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.920614    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df"} err="failed to get container status \"339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df\": rpc error: code = NotFound desc = could not find container \"339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df\": container with ID starting with 339aa91aa1545f344460fffd5a91c23f86bd85a255820ca0af3d52f63820b4df not found: ID does not exist"
Jul 20 10:55:45.920655 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.920631    2561 scope.go:117] "RemoveContainer" containerID="8f00f79f9f39a5b132a5b1d3955d29f551e686acd4ebb2f2365f80b48b39df05"
Jul 20 10:55:45.920845 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:55:45.920828    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f00f79f9f39a5b132a5b1d3955d29f551e686acd4ebb2f2365f80b48b39df05\": container with ID starting with 8f00f79f9f39a5b132a5b1d3955d29f551e686acd4ebb2f2365f80b48b39df05 not found: ID does not exist" containerID="8f00f79f9f39a5b132a5b1d3955d29f551e686acd4ebb2f2365f80b48b39df05"
Jul 20 10:55:45.920898 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.920852    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f00f79f9f39a5b132a5b1d3955d29f551e686acd4ebb2f2365f80b48b39df05"} err="failed to get container status \"8f00f79f9f39a5b132a5b1d3955d29f551e686acd4ebb2f2365f80b48b39df05\": rpc error: code = NotFound desc = could not find container \"8f00f79f9f39a5b132a5b1d3955d29f551e686acd4ebb2f2365f80b48b39df05\": container with ID starting with 8f00f79f9f39a5b132a5b1d3955d29f551e686acd4ebb2f2365f80b48b39df05 not found: ID does not exist"
Jul 20 10:55:45.924052 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:45.924034    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-k92jb"]
Jul 20 10:55:46.765292 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:46.765262    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5f1c982-dacb-4683-a183-536f41908bff" path="/var/lib/kubelet/pods/b5f1c982-dacb-4683-a183-536f41908bff/volumes"
Jul 20 10:55:53.381100 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.381056    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"]
Jul 20 10:55:53.381650 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.381634    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="storage-initializer"
Jul 20 10:55:53.381741 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.381654    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="storage-initializer"
Jul 20 10:55:53.381741 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.381671    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kube-rbac-proxy"
Jul 20 10:55:53.381741 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.381681    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kube-rbac-proxy"
Jul 20 10:55:53.381741 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.381691    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kserve-container"
Jul 20 10:55:53.381741 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.381700    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kserve-container"
Jul 20 10:55:53.382002 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.381765    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kserve-container"
Jul 20 10:55:53.382002 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.381776    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f1c982-dacb-4683-a183-536f41908bff" containerName="kube-rbac-proxy"
Jul 20 10:55:53.384688 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.384663    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.387471 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.387435    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 20 10:55:53.387760 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.387741    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:55:53.387861 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.387803    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:55:53.388875 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.388854    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:55:53.388975 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.388915    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 20 10:55:53.388975 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.388935    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:55:53.389079 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.388858    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:55:53.394499 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.394479    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"]
Jul 20 10:55:53.448509 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.448479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-proxy-tls\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.448509 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.448509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kserve-provision-location\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.448668 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.448542    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwck9\" (UniqueName: \"kubernetes.io/projected/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kube-api-access-rwck9\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.448668 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.448610    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.448668 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.448664    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-cabundle-cert\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.549442 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.549405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-proxy-tls\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.549442 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.549446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kserve-provision-location\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.549642 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.549480    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwck9\" (UniqueName: \"kubernetes.io/projected/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kube-api-access-rwck9\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.549642 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.549518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.549642 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.549535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-cabundle-cert\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.549642 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:55:53.549563    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Jul 20 10:55:53.549642 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:55:53.549639    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-proxy-tls podName:05d5e5f4-aa68-45f4-89e8-a6b6af24916f nodeName:}" failed. No retries permitted until 2026-07-20 10:55:54.049618555 +0000 UTC m=+3533.887717118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-proxy-tls") pod "isvc-sklearn-predictor-6fdcbb68cb-ct4qd" (UID: "05d5e5f4-aa68-45f4-89e8-a6b6af24916f") : secret "isvc-sklearn-predictor-serving-cert" not found
Jul 20 10:55:53.549920 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.549868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kserve-provision-location\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.550320 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.550301    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.550379 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.550301    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-cabundle-cert\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:53.561755 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:53.561737    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwck9\" (UniqueName: \"kubernetes.io/projected/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kube-api-access-rwck9\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:54.053969 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:54.053938    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-proxy-tls\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:54.056287 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:54.056268    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-proxy-tls\") pod \"isvc-sklearn-predictor-6fdcbb68cb-ct4qd\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:54.294970 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:54.294918    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:54.417723 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:54.417575    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"]
Jul 20 10:55:54.420407 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:55:54.420380    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d5e5f4_aa68_45f4_89e8_a6b6af24916f.slice/crio-249d9cf389da3c11afacc598a90b9f8ceb2449ad284ff4e0da239107840c373c WatchSource:0}: Error finding container 249d9cf389da3c11afacc598a90b9f8ceb2449ad284ff4e0da239107840c373c: Status 404 returned error can't find the container with id 249d9cf389da3c11afacc598a90b9f8ceb2449ad284ff4e0da239107840c373c
Jul 20 10:55:54.932996 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:54.932958    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" event={"ID":"05d5e5f4-aa68-45f4-89e8-a6b6af24916f","Type":"ContainerStarted","Data":"a008d1e6f11ccf814fa686e6a0ae41f9a0877a62811d95a69131f06f73888243"}
Jul 20 10:55:54.933217 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:54.933026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" event={"ID":"05d5e5f4-aa68-45f4-89e8-a6b6af24916f","Type":"ContainerStarted","Data":"249d9cf389da3c11afacc598a90b9f8ceb2449ad284ff4e0da239107840c373c"}
Jul 20 10:55:58.947916 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:58.947887    2561 generic.go:356] "Generic (PLEG): container finished" podID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerID="a008d1e6f11ccf814fa686e6a0ae41f9a0877a62811d95a69131f06f73888243" exitCode=0
Jul 20 10:55:58.948286 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:58.947968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" event={"ID":"05d5e5f4-aa68-45f4-89e8-a6b6af24916f","Type":"ContainerDied","Data":"a008d1e6f11ccf814fa686e6a0ae41f9a0877a62811d95a69131f06f73888243"}
Jul 20 10:55:59.954064 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:59.954032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" event={"ID":"05d5e5f4-aa68-45f4-89e8-a6b6af24916f","Type":"ContainerStarted","Data":"b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003"}
Jul 20 10:55:59.954471 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:59.954088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" event={"ID":"05d5e5f4-aa68-45f4-89e8-a6b6af24916f","Type":"ContainerStarted","Data":"ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6"}
Jul 20 10:55:59.954471 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:59.954411    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:59.954560 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:59.954532    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:55:59.955612 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:59.955588    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 20 10:55:59.974177 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:55:59.974139    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podStartSLOduration=3.382517424 podStartE2EDuration="6.974104441s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.591587017s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:55:59.972758309 +0000 UTC m=+3539.810856910" watchObservedRunningTime="2026-07-20 10:55:59.974104441 +0000 UTC m=+3539.812203026"
Jul 20 10:56:00.958369 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:56:00.958328    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 20 10:56:01.961726 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:56:01.961686    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 20 10:56:06.966285 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:56:06.966258    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:56:06.966900 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:56:06.966873    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 20 10:56:16.967725 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:56:16.967684    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 20 10:56:26.967147 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:56:26.967089    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 20 10:56:36.966850 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:56:36.966807    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 20 10:56:46.966964 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:56:46.966924    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 20 10:56:56.967584 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:56:56.967503    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 20 10:57:06.967139 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:06.967085    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 20 10:57:16.185316 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:16.185289    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:57:16.191713 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:16.191691    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 10:57:16.967744 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:16.967720    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:57:23.541611 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:23.541578    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"]
Jul 20 10:57:23.542219 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:23.541903    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container" containerID="cri-o://ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6" gracePeriod=30
Jul 20 10:57:23.542219 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:23.541918    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kube-rbac-proxy" containerID="cri-o://b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003" gracePeriod=30
Jul 20 10:57:24.262610 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:24.262576    2561 generic.go:356] "Generic (PLEG): container finished" podID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerID="b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003" exitCode=2
Jul 20 10:57:24.262818 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:24.262623    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" event={"ID":"05d5e5f4-aa68-45f4-89e8-a6b6af24916f","Type":"ContainerDied","Data":"b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003"}
Jul 20 10:57:26.962096 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:26.962049    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Jul 20 10:57:26.967394 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:26.967363    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 20 10:57:27.779791 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.779768    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:57:27.826543 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.826472    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kserve-provision-location\") pod \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") "
Jul 20 10:57:27.826543 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.826525    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-cabundle-cert\") pod \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") "
Jul 20 10:57:27.826543 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.826552    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwck9\" (UniqueName: \"kubernetes.io/projected/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kube-api-access-rwck9\") pod \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") "
Jul 20 10:57:27.826802 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.826585    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kube-rbac-proxy-sar-config\") pod \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") "
Jul 20 10:57:27.826802 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.826624    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-proxy-tls\") pod \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\" (UID: \"05d5e5f4-aa68-45f4-89e8-a6b6af24916f\") "
Jul 20 10:57:27.826909 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.826831    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "05d5e5f4-aa68-45f4-89e8-a6b6af24916f" (UID: "05d5e5f4-aa68-45f4-89e8-a6b6af24916f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:57:27.826970 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.826929    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "05d5e5f4-aa68-45f4-89e8-a6b6af24916f" (UID: "05d5e5f4-aa68-45f4-89e8-a6b6af24916f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:57:27.826970 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.826942    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "05d5e5f4-aa68-45f4-89e8-a6b6af24916f" (UID: "05d5e5f4-aa68-45f4-89e8-a6b6af24916f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:57:27.828606 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.828583    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kube-api-access-rwck9" (OuterVolumeSpecName: "kube-api-access-rwck9") pod "05d5e5f4-aa68-45f4-89e8-a6b6af24916f" (UID: "05d5e5f4-aa68-45f4-89e8-a6b6af24916f"). InnerVolumeSpecName "kube-api-access-rwck9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:57:27.828671 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.828649    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "05d5e5f4-aa68-45f4-89e8-a6b6af24916f" (UID: "05d5e5f4-aa68-45f4-89e8-a6b6af24916f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:57:27.927456 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.927433    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:57:27.927456 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.927455    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:57:27.927602 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.927464    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwck9\" (UniqueName: \"kubernetes.io/projected/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kube-api-access-rwck9\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:57:27.927602 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.927473    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:57:27.927602 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:27.927483    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05d5e5f4-aa68-45f4-89e8-a6b6af24916f-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:57:28.279467 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.279429    2561 generic.go:356] "Generic (PLEG): container finished" podID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerID="ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6" exitCode=0
Jul 20 10:57:28.279949 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.279496    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"
Jul 20 10:57:28.279949 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.279515    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" event={"ID":"05d5e5f4-aa68-45f4-89e8-a6b6af24916f","Type":"ContainerDied","Data":"ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6"}
Jul 20 10:57:28.279949 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.279593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd" event={"ID":"05d5e5f4-aa68-45f4-89e8-a6b6af24916f","Type":"ContainerDied","Data":"249d9cf389da3c11afacc598a90b9f8ceb2449ad284ff4e0da239107840c373c"}
Jul 20 10:57:28.279949 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.279640    2561 scope.go:117] "RemoveContainer" containerID="b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003"
Jul 20 10:57:28.287864 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.287847    2561 scope.go:117] "RemoveContainer" containerID="ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6"
Jul 20 10:57:28.294730 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.294712    2561 scope.go:117] "RemoveContainer" containerID="a008d1e6f11ccf814fa686e6a0ae41f9a0877a62811d95a69131f06f73888243"
Jul 20 10:57:28.300166 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.300144    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"]
Jul 20 10:57:28.302136 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.302094    2561 scope.go:117] "RemoveContainer" containerID="b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003"
Jul 20 10:57:28.302423 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:57:28.302398    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003\": container with ID starting with b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003 not found: ID does not exist" containerID="b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003"
Jul 20 10:57:28.302491 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.302435    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003"} err="failed to get container status \"b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003\": rpc error: code = NotFound desc = could not find container \"b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003\": container with ID starting with b981210b3c340f7ac260255c7b65dc0a6d7810e7bcb2bcedf51f693c9e71c003 not found: ID does not exist"
Jul 20 10:57:28.302491 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.302459    2561 scope.go:117] "RemoveContainer" containerID="ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6"
Jul 20 10:57:28.302721 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:57:28.302704    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6\": container with ID starting with ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6 not found: ID does not exist" containerID="ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6"
Jul 20 10:57:28.302765 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.302729    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6"} err="failed to get container status \"ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6\": rpc error: code = NotFound desc = could not find container \"ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6\": container with ID starting with ae2f07c2db79ba27c10af67490cbbc12c0ab3bbdca92536720a3f9e293d28fa6 not found: ID does not exist"
Jul 20 10:57:28.302765 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.302744    2561 scope.go:117] "RemoveContainer" containerID="a008d1e6f11ccf814fa686e6a0ae41f9a0877a62811d95a69131f06f73888243"
Jul 20 10:57:28.303008 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:57:28.302985    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a008d1e6f11ccf814fa686e6a0ae41f9a0877a62811d95a69131f06f73888243\": container with ID starting with a008d1e6f11ccf814fa686e6a0ae41f9a0877a62811d95a69131f06f73888243 not found: ID does not exist" containerID="a008d1e6f11ccf814fa686e6a0ae41f9a0877a62811d95a69131f06f73888243"
Jul 20 10:57:28.303094 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.303013    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a008d1e6f11ccf814fa686e6a0ae41f9a0877a62811d95a69131f06f73888243"} err="failed to get container status \"a008d1e6f11ccf814fa686e6a0ae41f9a0877a62811d95a69131f06f73888243\": rpc error: code = NotFound desc = could not find container \"a008d1e6f11ccf814fa686e6a0ae41f9a0877a62811d95a69131f06f73888243\": container with ID starting with a008d1e6f11ccf814fa686e6a0ae41f9a0877a62811d95a69131f06f73888243 not found: ID does not exist"
Jul 20 10:57:28.303199 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.303182    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6fdcbb68cb-ct4qd"]
Jul 20 10:57:28.734824 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.734789    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"]
Jul 20 10:57:28.735067 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.735053    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container"
Jul 20 10:57:28.735067 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.735066    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container"
Jul 20 10:57:28.735220 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.735085    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kube-rbac-proxy"
Jul 20 10:57:28.735220 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.735091    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kube-rbac-proxy"
Jul 20 10:57:28.735220 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.735098    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="storage-initializer"
Jul 20 10:57:28.735220 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.735103    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="storage-initializer"
Jul 20 10:57:28.735220 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.735196    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kserve-container"
Jul 20 10:57:28.735220 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.735209    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" containerName="kube-rbac-proxy"
Jul 20 10:57:28.739832 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.739812    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.742562 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.742540    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 20 10:57:28.742690 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.742577    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:57:28.742690 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.742576    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:57:28.742909 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.742895    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:57:28.743076 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.743063    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:57:28.743270 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.743258    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:57:28.743320 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.743259    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 20 10:57:28.748787 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.748768    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"]
Jul 20 10:57:28.765471 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.765440    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05d5e5f4-aa68-45f4-89e8-a6b6af24916f" path="/var/lib/kubelet/pods/05d5e5f4-aa68-45f4-89e8-a6b6af24916f/volumes"
Jul 20 10:57:28.834158 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.834134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gfmw\" (UniqueName: \"kubernetes.io/projected/015179d1-a3b1-4448-afe5-d24cf65b5e84-kube-api-access-4gfmw\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.834283 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.834177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/015179d1-a3b1-4448-afe5-d24cf65b5e84-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.834283 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.834217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/015179d1-a3b1-4448-afe5-d24cf65b5e84-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.834283 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.834251    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/015179d1-a3b1-4448-afe5-d24cf65b5e84-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.834464 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.834324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/015179d1-a3b1-4448-afe5-d24cf65b5e84-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.935003 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.934977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfmw\" (UniqueName: \"kubernetes.io/projected/015179d1-a3b1-4448-afe5-d24cf65b5e84-kube-api-access-4gfmw\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.935368 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.935014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/015179d1-a3b1-4448-afe5-d24cf65b5e84-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.935368 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.935051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/015179d1-a3b1-4448-afe5-d24cf65b5e84-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.935368 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.935082    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/015179d1-a3b1-4448-afe5-d24cf65b5e84-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.935368 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.935141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/015179d1-a3b1-4448-afe5-d24cf65b5e84-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.935616 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.935582    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/015179d1-a3b1-4448-afe5-d24cf65b5e84-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.935747 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.935728    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/015179d1-a3b1-4448-afe5-d24cf65b5e84-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.935818 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.935800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/015179d1-a3b1-4448-afe5-d24cf65b5e84-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.937701 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.937680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/015179d1-a3b1-4448-afe5-d24cf65b5e84-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:28.943948 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:28.943928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gfmw\" (UniqueName: \"kubernetes.io/projected/015179d1-a3b1-4448-afe5-d24cf65b5e84-kube-api-access-4gfmw\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:29.050849 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:29.050779    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:29.178508 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:29.178485    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"]
Jul 20 10:57:29.183635 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:57:29.183560    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015179d1_a3b1_4448_afe5_d24cf65b5e84.slice/crio-3efedb365c02ebbb0fd556fc2bc44fc7605bdea183ddc19d6d87405326438cde WatchSource:0}: Error finding container 3efedb365c02ebbb0fd556fc2bc44fc7605bdea183ddc19d6d87405326438cde: Status 404 returned error can't find the container with id 3efedb365c02ebbb0fd556fc2bc44fc7605bdea183ddc19d6d87405326438cde
Jul 20 10:57:29.285450 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:29.285414    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" event={"ID":"015179d1-a3b1-4448-afe5-d24cf65b5e84","Type":"ContainerStarted","Data":"4ac9e61c48273ed0b00f8e9f6c09ca1592b5047ed24c1b902fad980f484003a2"}
Jul 20 10:57:29.285923 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:29.285480    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" event={"ID":"015179d1-a3b1-4448-afe5-d24cf65b5e84","Type":"ContainerStarted","Data":"3efedb365c02ebbb0fd556fc2bc44fc7605bdea183ddc19d6d87405326438cde"}
Jul 20 10:57:33.301022 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:33.300942    2561 generic.go:356] "Generic (PLEG): container finished" podID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerID="4ac9e61c48273ed0b00f8e9f6c09ca1592b5047ed24c1b902fad980f484003a2" exitCode=0
Jul 20 10:57:33.301396 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:33.301015    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" event={"ID":"015179d1-a3b1-4448-afe5-d24cf65b5e84","Type":"ContainerDied","Data":"4ac9e61c48273ed0b00f8e9f6c09ca1592b5047ed24c1b902fad980f484003a2"}
Jul 20 10:57:34.307159 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:34.307106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" event={"ID":"015179d1-a3b1-4448-afe5-d24cf65b5e84","Type":"ContainerStarted","Data":"c385a1bf4024e4cce893b65a4ce5404397fb32baa410c0f9d0677e4b1a22722d"}
Jul 20 10:57:34.307558 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:34.307183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" event={"ID":"015179d1-a3b1-4448-afe5-d24cf65b5e84","Type":"ContainerStarted","Data":"e8f1ee6e294ed8a724f7d5ac03033312507dcd848f7cedab324383b0e7f17ce7"}
Jul 20 10:57:34.307558 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:34.307373    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:34.331988 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:34.331940    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" podStartSLOduration=2.666014264 podStartE2EDuration="6.331924813s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.665910549s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:57:34.32980997 +0000 UTC m=+3634.167908599" watchObservedRunningTime="2026-07-20 10:57:34.331924813 +0000 UTC m=+3634.170023398"
Jul 20 10:57:35.310595 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:35.310563    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:57:41.319317 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:57:41.319288    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:58:11.355879 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:11.355837    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 20 10:58:21.322376 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:21.322344    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:58:28.874385 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:28.874345    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"]
Jul 20 10:58:28.874799 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:28.874616    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerName="kube-rbac-proxy" containerID="cri-o://c385a1bf4024e4cce893b65a4ce5404397fb32baa410c0f9d0677e4b1a22722d" gracePeriod=30
Jul 20 10:58:28.874799 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:28.874574    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerName="kserve-container" containerID="cri-o://e8f1ee6e294ed8a724f7d5ac03033312507dcd848f7cedab324383b0e7f17ce7" gracePeriod=30
Jul 20 10:58:29.509087 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:29.509056    2561 generic.go:356] "Generic (PLEG): container finished" podID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerID="c385a1bf4024e4cce893b65a4ce5404397fb32baa410c0f9d0677e4b1a22722d" exitCode=2
Jul 20 10:58:29.509284 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:29.509106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" event={"ID":"015179d1-a3b1-4448-afe5-d24cf65b5e84","Type":"ContainerDied","Data":"c385a1bf4024e4cce893b65a4ce5404397fb32baa410c0f9d0677e4b1a22722d"}
Jul 20 10:58:31.314649 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:31.314599    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Jul 20 10:58:34.114492 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.114453    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"]
Jul 20 10:58:34.118160 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.118133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.121466 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.121446    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 20 10:58:34.122369 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.122349    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 20 10:58:34.132912 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.132880    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"]
Jul 20 10:58:34.191910 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.191881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.192090 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.191922    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2m5w\" (UniqueName: \"kubernetes.io/projected/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kube-api-access-x2m5w\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.192090 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.191944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.192090 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.191970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.192090 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.192034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.293227 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.293191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.293417 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.293247    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.293417 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.293294    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2m5w\" (UniqueName: \"kubernetes.io/projected/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kube-api-access-x2m5w\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.293417 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.293324    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.293417 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.293349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.293708 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.293682    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.293985 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.293963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.294038 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.293986    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.295723 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.295702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.309735 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.309712    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2m5w\" (UniqueName: \"kubernetes.io/projected/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kube-api-access-x2m5w\") pod \"isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.427662 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.427594    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:34.560170 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:34.560139    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"]
Jul 20 10:58:34.563049 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:58:34.563023    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96b2cc5_ff99_451e_89a5_0f0b35cf4ba9.slice/crio-4b2a489c2db5546a7931b474900b993e80d459e4a6697c7ada6796731bb0f75a WatchSource:0}: Error finding container 4b2a489c2db5546a7931b474900b993e80d459e4a6697c7ada6796731bb0f75a: Status 404 returned error can't find the container with id 4b2a489c2db5546a7931b474900b993e80d459e4a6697c7ada6796731bb0f75a
Jul 20 10:58:35.533655 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.533629    2561 generic.go:356] "Generic (PLEG): container finished" podID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerID="e8f1ee6e294ed8a724f7d5ac03033312507dcd848f7cedab324383b0e7f17ce7" exitCode=0
Jul 20 10:58:35.534022 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.533695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" event={"ID":"015179d1-a3b1-4448-afe5-d24cf65b5e84","Type":"ContainerDied","Data":"e8f1ee6e294ed8a724f7d5ac03033312507dcd848f7cedab324383b0e7f17ce7"}
Jul 20 10:58:35.535285 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.535257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" event={"ID":"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9","Type":"ContainerStarted","Data":"a3e08729ee05dba644fc97590b4048e040bdfaa375ff864dc7d02895d435495f"}
Jul 20 10:58:35.535438 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.535301    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" event={"ID":"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9","Type":"ContainerStarted","Data":"4b2a489c2db5546a7931b474900b993e80d459e4a6697c7ada6796731bb0f75a"}
Jul 20 10:58:35.616961 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.616937    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:58:35.704049 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.704014    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/015179d1-a3b1-4448-afe5-d24cf65b5e84-proxy-tls\") pod \"015179d1-a3b1-4448-afe5-d24cf65b5e84\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") "
Jul 20 10:58:35.704049 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.704054    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/015179d1-a3b1-4448-afe5-d24cf65b5e84-kserve-provision-location\") pod \"015179d1-a3b1-4448-afe5-d24cf65b5e84\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") "
Jul 20 10:58:35.704335 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.704084    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/015179d1-a3b1-4448-afe5-d24cf65b5e84-cabundle-cert\") pod \"015179d1-a3b1-4448-afe5-d24cf65b5e84\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") "
Jul 20 10:58:35.704335 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.704136    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gfmw\" (UniqueName: \"kubernetes.io/projected/015179d1-a3b1-4448-afe5-d24cf65b5e84-kube-api-access-4gfmw\") pod \"015179d1-a3b1-4448-afe5-d24cf65b5e84\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") "
Jul 20 10:58:35.704335 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.704186    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/015179d1-a3b1-4448-afe5-d24cf65b5e84-kube-rbac-proxy-sar-config\") pod \"015179d1-a3b1-4448-afe5-d24cf65b5e84\" (UID: \"015179d1-a3b1-4448-afe5-d24cf65b5e84\") "
Jul 20 10:58:35.704504 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.704475    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015179d1-a3b1-4448-afe5-d24cf65b5e84-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "015179d1-a3b1-4448-afe5-d24cf65b5e84" (UID: "015179d1-a3b1-4448-afe5-d24cf65b5e84"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:58:35.704564 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.704495    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015179d1-a3b1-4448-afe5-d24cf65b5e84-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "015179d1-a3b1-4448-afe5-d24cf65b5e84" (UID: "015179d1-a3b1-4448-afe5-d24cf65b5e84"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:58:35.704564 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.704546    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015179d1-a3b1-4448-afe5-d24cf65b5e84-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "015179d1-a3b1-4448-afe5-d24cf65b5e84" (UID: "015179d1-a3b1-4448-afe5-d24cf65b5e84"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:58:35.706196 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.706172    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015179d1-a3b1-4448-afe5-d24cf65b5e84-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "015179d1-a3b1-4448-afe5-d24cf65b5e84" (UID: "015179d1-a3b1-4448-afe5-d24cf65b5e84"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:58:35.706306 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.706208    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015179d1-a3b1-4448-afe5-d24cf65b5e84-kube-api-access-4gfmw" (OuterVolumeSpecName: "kube-api-access-4gfmw") pod "015179d1-a3b1-4448-afe5-d24cf65b5e84" (UID: "015179d1-a3b1-4448-afe5-d24cf65b5e84"). InnerVolumeSpecName "kube-api-access-4gfmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:58:35.805071 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.804982    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/015179d1-a3b1-4448-afe5-d24cf65b5e84-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:58:35.805071 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.805013    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/015179d1-a3b1-4448-afe5-d24cf65b5e84-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:58:35.805071 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.805024    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/015179d1-a3b1-4448-afe5-d24cf65b5e84-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:58:35.805071 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.805033    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/015179d1-a3b1-4448-afe5-d24cf65b5e84-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:58:35.805071 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:35.805043    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gfmw\" (UniqueName: \"kubernetes.io/projected/015179d1-a3b1-4448-afe5-d24cf65b5e84-kube-api-access-4gfmw\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:58:36.540920 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:36.540880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj" event={"ID":"015179d1-a3b1-4448-afe5-d24cf65b5e84","Type":"ContainerDied","Data":"3efedb365c02ebbb0fd556fc2bc44fc7605bdea183ddc19d6d87405326438cde"}
Jul 20 10:58:36.540920 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:36.540918    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"
Jul 20 10:58:36.541426 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:36.540968    2561 scope.go:117] "RemoveContainer" containerID="c385a1bf4024e4cce893b65a4ce5404397fb32baa410c0f9d0677e4b1a22722d"
Jul 20 10:58:36.549631 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:36.549610    2561 scope.go:117] "RemoveContainer" containerID="e8f1ee6e294ed8a724f7d5ac03033312507dcd848f7cedab324383b0e7f17ce7"
Jul 20 10:58:36.556634 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:36.556618    2561 scope.go:117] "RemoveContainer" containerID="4ac9e61c48273ed0b00f8e9f6c09ca1592b5047ed24c1b902fad980f484003a2"
Jul 20 10:58:36.564756 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:36.564729    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"]
Jul 20 10:58:36.572528 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:36.572508    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-rsnhj"]
Jul 20 10:58:36.764747 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:36.764713    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" path="/var/lib/kubelet/pods/015179d1-a3b1-4448-afe5-d24cf65b5e84/volumes"
Jul 20 10:58:40.557197 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:40.557162    2561 generic.go:356] "Generic (PLEG): container finished" podID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerID="a3e08729ee05dba644fc97590b4048e040bdfaa375ff864dc7d02895d435495f" exitCode=0
Jul 20 10:58:40.557618 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:40.557235    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" event={"ID":"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9","Type":"ContainerDied","Data":"a3e08729ee05dba644fc97590b4048e040bdfaa375ff864dc7d02895d435495f"}
Jul 20 10:58:41.563259 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:41.563228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" event={"ID":"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9","Type":"ContainerStarted","Data":"2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086"}
Jul 20 10:58:41.563687 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:41.563292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" event={"ID":"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9","Type":"ContainerStarted","Data":"6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a"}
Jul 20 10:58:41.563687 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:41.563494    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:41.591171 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:41.591126    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" podStartSLOduration=2.114784321 podStartE2EDuration="7.591098495s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.476314174s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:58:41.589860587 +0000 UTC m=+3701.427959207" watchObservedRunningTime="2026-07-20 10:58:41.591098495 +0000 UTC m=+3701.429197080"
Jul 20 10:58:42.567124 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:42.567073    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:42.568569 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:42.568545    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Jul 20 10:58:43.570723 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:43.570678    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Jul 20 10:58:48.575903 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:48.575875    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:58:48.576406 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:48.576380    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Jul 20 10:58:58.576842 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:58:58.576808    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:59:10.940297 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:10.940264    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69_f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9/kserve-container/0.log"
Jul 20 10:59:11.093683 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.093647    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"]
Jul 20 10:59:11.093988 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.093923    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="kserve-container" containerID="cri-o://6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a" gracePeriod=30
Jul 20 10:59:11.094168 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.093960    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="kube-rbac-proxy" containerID="cri-o://2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086" gracePeriod=30
Jul 20 10:59:11.681576 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.681546    2561 generic.go:356] "Generic (PLEG): container finished" podID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerID="2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086" exitCode=2
Jul 20 10:59:11.681775 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.681622    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" event={"ID":"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9","Type":"ContainerDied","Data":"2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086"}
Jul 20 10:59:11.930883 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.930858    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:59:11.972796 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.972725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-cabundle-cert\") pod \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") "
Jul 20 10:59:11.972796 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.972771    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kserve-provision-location\") pod \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") "
Jul 20 10:59:11.973235 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.972812    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2m5w\" (UniqueName: \"kubernetes.io/projected/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kube-api-access-x2m5w\") pod \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") "
Jul 20 10:59:11.973235 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.972836    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kube-rbac-proxy-sar-config\") pod \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") "
Jul 20 10:59:11.973235 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.972870    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-proxy-tls\") pod \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\" (UID: \"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9\") "
Jul 20 10:59:11.973235 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.973082    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" (UID: "f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:59:11.973235 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.973215    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" (UID: "f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:59:11.974917 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.974894    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" (UID: "f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:59:11.975025 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.974995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kube-api-access-x2m5w" (OuterVolumeSpecName: "kube-api-access-x2m5w") pod "f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" (UID: "f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9"). InnerVolumeSpecName "kube-api-access-x2m5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:59:11.983483 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:11.983461    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" (UID: "f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:59:12.073400 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.073378    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:59:12.073400 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.073399    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:59:12.073536 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.073410    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2m5w\" (UniqueName: \"kubernetes.io/projected/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kube-api-access-x2m5w\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:59:12.073536 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.073419    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:59:12.073536 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.073429    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 10:59:12.686689 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.686652    2561 generic.go:356] "Generic (PLEG): container finished" podID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerID="6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a" exitCode=0
Jul 20 10:59:12.686898 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.686705    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" event={"ID":"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9","Type":"ContainerDied","Data":"6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a"}
Jul 20 10:59:12.686898 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.686731    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"
Jul 20 10:59:12.686898 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.686778    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69" event={"ID":"f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9","Type":"ContainerDied","Data":"4b2a489c2db5546a7931b474900b993e80d459e4a6697c7ada6796731bb0f75a"}
Jul 20 10:59:12.686898 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.686806    2561 scope.go:117] "RemoveContainer" containerID="2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086"
Jul 20 10:59:12.695027 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.695011    2561 scope.go:117] "RemoveContainer" containerID="6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a"
Jul 20 10:59:12.702044 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.702018    2561 scope.go:117] "RemoveContainer" containerID="a3e08729ee05dba644fc97590b4048e040bdfaa375ff864dc7d02895d435495f"
Jul 20 10:59:12.708768 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.708752    2561 scope.go:117] "RemoveContainer" containerID="2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086"
Jul 20 10:59:12.709033 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:59:12.709007    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086\": container with ID starting with 2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086 not found: ID does not exist" containerID="2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086"
Jul 20 10:59:12.709148 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.709039    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086"} err="failed to get container status \"2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086\": rpc error: code = NotFound desc = could not find container \"2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086\": container with ID starting with 2debf06bcdc781db72337cd7810544918fac8e87475d484a460a8a446f5ca086 not found: ID does not exist"
Jul 20 10:59:12.709148 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.709060    2561 scope.go:117] "RemoveContainer" containerID="6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a"
Jul 20 10:59:12.709636 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:59:12.709610    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a\": container with ID starting with 6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a not found: ID does not exist" containerID="6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a"
Jul 20 10:59:12.709762 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.709643    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a"} err="failed to get container status \"6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a\": rpc error: code = NotFound desc = could not find container \"6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a\": container with ID starting with 6ac7be57a00f5890d097eb352c96e6905f673e463205b55393dae286df2d801a not found: ID does not exist"
Jul 20 10:59:12.709762 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.709724    2561 scope.go:117] "RemoveContainer" containerID="a3e08729ee05dba644fc97590b4048e040bdfaa375ff864dc7d02895d435495f"
Jul 20 10:59:12.710053 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:59:12.710031    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e08729ee05dba644fc97590b4048e040bdfaa375ff864dc7d02895d435495f\": container with ID starting with a3e08729ee05dba644fc97590b4048e040bdfaa375ff864dc7d02895d435495f not found: ID does not exist" containerID="a3e08729ee05dba644fc97590b4048e040bdfaa375ff864dc7d02895d435495f"
Jul 20 10:59:12.710165 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.710056    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e08729ee05dba644fc97590b4048e040bdfaa375ff864dc7d02895d435495f"} err="failed to get container status \"a3e08729ee05dba644fc97590b4048e040bdfaa375ff864dc7d02895d435495f\": rpc error: code = NotFound desc = could not find container \"a3e08729ee05dba644fc97590b4048e040bdfaa375ff864dc7d02895d435495f\": container with ID starting with a3e08729ee05dba644fc97590b4048e040bdfaa375ff864dc7d02895d435495f not found: ID does not exist"
Jul 20 10:59:12.711461 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.711442    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"]
Jul 20 10:59:12.715106 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.715083    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5cbb8cfcd4-wgm69"]
Jul 20 10:59:12.765201 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:12.765180    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" path="/var/lib/kubelet/pods/f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9/volumes"
Jul 20 10:59:16.242509 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242471    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"]
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242790    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerName="storage-initializer"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242804    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerName="storage-initializer"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242812    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="storage-initializer"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242819    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="storage-initializer"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242830    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerName="kserve-container"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242839    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerName="kserve-container"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242846    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="kube-rbac-proxy"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242853    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="kube-rbac-proxy"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242863    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="kserve-container"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242868    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="kserve-container"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242875    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerName="kube-rbac-proxy"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242883    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerName="kube-rbac-proxy"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242947    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerName="kserve-container"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242958    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="kserve-container"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242968    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f96b2cc5-ff99-451e-89a5-0f0b35cf4ba9" containerName="kube-rbac-proxy"
Jul 20 10:59:16.242976 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.242982    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="015179d1-a3b1-4448-afe5-d24cf65b5e84" containerName="kube-rbac-proxy"
Jul 20 10:59:16.247482 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.247466    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.250382 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.250360    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 10:59:16.250522 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.250393    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:59:16.250522 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.250401    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:59:16.250522 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.250454    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:59:16.250522 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.250460    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 20 10:59:16.251337 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.251321    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 20 10:59:16.251453 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.251390    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:59:16.256982 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.256960    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"]
Jul 20 10:59:16.305266 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.305244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b36406ce-a27c-491f-b0a3-aaa1346aba68-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.305382 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.305279    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr56f\" (UniqueName: \"kubernetes.io/projected/b36406ce-a27c-491f-b0a3-aaa1346aba68-kube-api-access-vr56f\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.305382 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.305313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b36406ce-a27c-491f-b0a3-aaa1346aba68-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.305382 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.305365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b36406ce-a27c-491f-b0a3-aaa1346aba68-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.305493 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.305383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b36406ce-a27c-491f-b0a3-aaa1346aba68-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.406259 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.406224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b36406ce-a27c-491f-b0a3-aaa1346aba68-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.406427 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.406284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b36406ce-a27c-491f-b0a3-aaa1346aba68-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.406427 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.406314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr56f\" (UniqueName: \"kubernetes.io/projected/b36406ce-a27c-491f-b0a3-aaa1346aba68-kube-api-access-vr56f\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.406427 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.406355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b36406ce-a27c-491f-b0a3-aaa1346aba68-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.406604 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:59:16.406473    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 20 10:59:16.406604 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.406488    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b36406ce-a27c-491f-b0a3-aaa1346aba68-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.406604 ip-10-0-134-80 kubenswrapper[2561]: E0720 10:59:16.406547    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36406ce-a27c-491f-b0a3-aaa1346aba68-proxy-tls podName:b36406ce-a27c-491f-b0a3-aaa1346aba68 nodeName:}" failed. No retries permitted until 2026-07-20 10:59:16.906524796 +0000 UTC m=+3736.744623360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b36406ce-a27c-491f-b0a3-aaa1346aba68-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" (UID: "b36406ce-a27c-491f-b0a3-aaa1346aba68") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 20 10:59:16.406898 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.406879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b36406ce-a27c-491f-b0a3-aaa1346aba68-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.406992 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.406976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b36406ce-a27c-491f-b0a3-aaa1346aba68-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.407090 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.407074    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b36406ce-a27c-491f-b0a3-aaa1346aba68-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.417354 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.417333    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr56f\" (UniqueName: \"kubernetes.io/projected/b36406ce-a27c-491f-b0a3-aaa1346aba68-kube-api-access-vr56f\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.909720 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.909688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b36406ce-a27c-491f-b0a3-aaa1346aba68-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:16.912038 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:16.912019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b36406ce-a27c-491f-b0a3-aaa1346aba68-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:17.157318 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:17.157283    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:17.288634 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:17.288606    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"]
Jul 20 10:59:17.292058 ip-10-0-134-80 kubenswrapper[2561]: W0720 10:59:17.292022    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36406ce_a27c_491f_b0a3_aaa1346aba68.slice/crio-bd38e4dc0ef167c1bec411e97fb1b5807f917208d14e175c8b99087508ea0f95 WatchSource:0}: Error finding container bd38e4dc0ef167c1bec411e97fb1b5807f917208d14e175c8b99087508ea0f95: Status 404 returned error can't find the container with id bd38e4dc0ef167c1bec411e97fb1b5807f917208d14e175c8b99087508ea0f95
Jul 20 10:59:17.705756 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:17.705717    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" event={"ID":"b36406ce-a27c-491f-b0a3-aaa1346aba68","Type":"ContainerStarted","Data":"42aecee09052c0a188fb42cb2fe81ffa521e57dcc534e248dd371664d7fefd66"}
Jul 20 10:59:17.705940 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:17.705792    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" event={"ID":"b36406ce-a27c-491f-b0a3-aaa1346aba68","Type":"ContainerStarted","Data":"bd38e4dc0ef167c1bec411e97fb1b5807f917208d14e175c8b99087508ea0f95"}
Jul 20 10:59:21.721011 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:21.720977    2561 generic.go:356] "Generic (PLEG): container finished" podID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerID="42aecee09052c0a188fb42cb2fe81ffa521e57dcc534e248dd371664d7fefd66" exitCode=0
Jul 20 10:59:21.721455 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:21.721048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" event={"ID":"b36406ce-a27c-491f-b0a3-aaa1346aba68","Type":"ContainerDied","Data":"42aecee09052c0a188fb42cb2fe81ffa521e57dcc534e248dd371664d7fefd66"}
Jul 20 10:59:22.726256 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:22.726220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" event={"ID":"b36406ce-a27c-491f-b0a3-aaa1346aba68","Type":"ContainerStarted","Data":"b3be490932f844567cd7da1c4837bdb635bfb6d48be59b8895e45820b540ae84"}
Jul 20 10:59:22.726680 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:22.726279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" event={"ID":"b36406ce-a27c-491f-b0a3-aaa1346aba68","Type":"ContainerStarted","Data":"3f04b22847366b3974ed26899d01467904673043eae6f9eba972e0ac5474e21c"}
Jul 20 10:59:22.726680 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:22.726487    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:22.726680 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:22.726660    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:22.749073 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:22.749029    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" podStartSLOduration=3.085825106 podStartE2EDuration="6.749014972s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.663189866s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:59:22.748526365 +0000 UTC m=+3742.586624983" watchObservedRunningTime="2026-07-20 10:59:22.749014972 +0000 UTC m=+3742.587113556"
Jul 20 10:59:28.736494 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:28.736465    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 10:59:58.756467 ip-10-0-134-80 kubenswrapper[2561]: I0720 10:59:58.756382    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 20 11:00:08.739329 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:08.739297    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 11:00:16.359346 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:16.359312    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"]
Jul 20 11:00:16.359906 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:16.359572    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerName="kserve-container" containerID="cri-o://3f04b22847366b3974ed26899d01467904673043eae6f9eba972e0ac5474e21c" gracePeriod=30
Jul 20 11:00:16.359906 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:16.359598    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerName="kube-rbac-proxy" containerID="cri-o://b3be490932f844567cd7da1c4837bdb635bfb6d48be59b8895e45820b540ae84" gracePeriod=30
Jul 20 11:00:16.931798 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:16.931762    2561 generic.go:356] "Generic (PLEG): container finished" podID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerID="b3be490932f844567cd7da1c4837bdb635bfb6d48be59b8895e45820b540ae84" exitCode=2
Jul 20 11:00:16.931961 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:16.931835    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" event={"ID":"b36406ce-a27c-491f-b0a3-aaa1346aba68","Type":"ContainerDied","Data":"b3be490932f844567cd7da1c4837bdb635bfb6d48be59b8895e45820b540ae84"}
Jul 20 11:00:18.730763 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:18.730719    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Jul 20 11:00:21.514659 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.514627    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"]
Jul 20 11:00:21.517767 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.517748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.520132 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.520083    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 20 11:00:21.520229 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.520087    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 20 11:00:21.529011 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.528990    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"]
Jul 20 11:00:21.668881 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.668856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a338166e-9993-4092-a178-98130d596df8-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.669065 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.668899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a338166e-9993-4092-a178-98130d596df8-proxy-tls\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.669065 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.668933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9rc9\" (UniqueName: \"kubernetes.io/projected/a338166e-9993-4092-a178-98130d596df8-kube-api-access-h9rc9\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.669065 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.668970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a338166e-9993-4092-a178-98130d596df8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.669065 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.668989    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a338166e-9993-4092-a178-98130d596df8-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.769778 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.769688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a338166e-9993-4092-a178-98130d596df8-proxy-tls\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.769778 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.769746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9rc9\" (UniqueName: \"kubernetes.io/projected/a338166e-9993-4092-a178-98130d596df8-kube-api-access-h9rc9\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.770168 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.769782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a338166e-9993-4092-a178-98130d596df8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.770168 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.769813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a338166e-9993-4092-a178-98130d596df8-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.770168 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.769846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a338166e-9993-4092-a178-98130d596df8-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.770503 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.770483    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a338166e-9993-4092-a178-98130d596df8-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.770659 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.770634    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a338166e-9993-4092-a178-98130d596df8-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.770725 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.770639    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a338166e-9993-4092-a178-98130d596df8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.772393 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.772373    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a338166e-9993-4092-a178-98130d596df8-proxy-tls\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.779011 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.778988    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9rc9\" (UniqueName: \"kubernetes.io/projected/a338166e-9993-4092-a178-98130d596df8-kube-api-access-h9rc9\") pod \"isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.828516 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.828494    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:21.954506 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:21.954475    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"]
Jul 20 11:00:21.957348 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:00:21.957312    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda338166e_9993_4092_a178_98130d596df8.slice/crio-8cb203182bf0481ce206d9f0ba8a12f56e8a44ffa8378c6d85bf2863427ebc91 WatchSource:0}: Error finding container 8cb203182bf0481ce206d9f0ba8a12f56e8a44ffa8378c6d85bf2863427ebc91: Status 404 returned error can't find the container with id 8cb203182bf0481ce206d9f0ba8a12f56e8a44ffa8378c6d85bf2863427ebc91
Jul 20 11:00:22.957610 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:22.957575    2561 generic.go:356] "Generic (PLEG): container finished" podID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerID="3f04b22847366b3974ed26899d01467904673043eae6f9eba972e0ac5474e21c" exitCode=0
Jul 20 11:00:22.957968 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:22.957641    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" event={"ID":"b36406ce-a27c-491f-b0a3-aaa1346aba68","Type":"ContainerDied","Data":"3f04b22847366b3974ed26899d01467904673043eae6f9eba972e0ac5474e21c"}
Jul 20 11:00:22.959247 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:22.959225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" event={"ID":"a338166e-9993-4092-a178-98130d596df8","Type":"ContainerStarted","Data":"e7d8c4f2a313caad710391ddd705857f5befdd3fd8aecb163dfde9f220709acc"}
Jul 20 11:00:22.959367 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:22.959270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" event={"ID":"a338166e-9993-4092-a178-98130d596df8","Type":"ContainerStarted","Data":"8cb203182bf0481ce206d9f0ba8a12f56e8a44ffa8378c6d85bf2863427ebc91"}
Jul 20 11:00:23.006756 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.006737    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 11:00:23.180773 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.180684    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr56f\" (UniqueName: \"kubernetes.io/projected/b36406ce-a27c-491f-b0a3-aaa1346aba68-kube-api-access-vr56f\") pod \"b36406ce-a27c-491f-b0a3-aaa1346aba68\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") "
Jul 20 11:00:23.180773 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.180733    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b36406ce-a27c-491f-b0a3-aaa1346aba68-kserve-provision-location\") pod \"b36406ce-a27c-491f-b0a3-aaa1346aba68\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") "
Jul 20 11:00:23.181016 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.180787    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b36406ce-a27c-491f-b0a3-aaa1346aba68-cabundle-cert\") pod \"b36406ce-a27c-491f-b0a3-aaa1346aba68\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") "
Jul 20 11:00:23.181016 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.180809    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b36406ce-a27c-491f-b0a3-aaa1346aba68-proxy-tls\") pod \"b36406ce-a27c-491f-b0a3-aaa1346aba68\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") "
Jul 20 11:00:23.181016 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.180832    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b36406ce-a27c-491f-b0a3-aaa1346aba68-kube-rbac-proxy-sar-config\") pod \"b36406ce-a27c-491f-b0a3-aaa1346aba68\" (UID: \"b36406ce-a27c-491f-b0a3-aaa1346aba68\") "
Jul 20 11:00:23.181208 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.181133    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36406ce-a27c-491f-b0a3-aaa1346aba68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b36406ce-a27c-491f-b0a3-aaa1346aba68" (UID: "b36406ce-a27c-491f-b0a3-aaa1346aba68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:00:23.181208 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.181174    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36406ce-a27c-491f-b0a3-aaa1346aba68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b36406ce-a27c-491f-b0a3-aaa1346aba68" (UID: "b36406ce-a27c-491f-b0a3-aaa1346aba68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:00:23.181289 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.181227    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36406ce-a27c-491f-b0a3-aaa1346aba68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b36406ce-a27c-491f-b0a3-aaa1346aba68" (UID: "b36406ce-a27c-491f-b0a3-aaa1346aba68"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:00:23.182739 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.182717    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36406ce-a27c-491f-b0a3-aaa1346aba68-kube-api-access-vr56f" (OuterVolumeSpecName: "kube-api-access-vr56f") pod "b36406ce-a27c-491f-b0a3-aaa1346aba68" (UID: "b36406ce-a27c-491f-b0a3-aaa1346aba68"). InnerVolumeSpecName "kube-api-access-vr56f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:00:23.182835 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.182816    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36406ce-a27c-491f-b0a3-aaa1346aba68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b36406ce-a27c-491f-b0a3-aaa1346aba68" (UID: "b36406ce-a27c-491f-b0a3-aaa1346aba68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:00:23.281959 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.281926    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b36406ce-a27c-491f-b0a3-aaa1346aba68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:00:23.281959 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.281953    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr56f\" (UniqueName: \"kubernetes.io/projected/b36406ce-a27c-491f-b0a3-aaa1346aba68-kube-api-access-vr56f\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:00:23.281959 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.281963    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b36406ce-a27c-491f-b0a3-aaa1346aba68-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:00:23.282204 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.281971    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b36406ce-a27c-491f-b0a3-aaa1346aba68-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:00:23.282204 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.281980    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b36406ce-a27c-491f-b0a3-aaa1346aba68-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:00:23.965123 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.965079    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"
Jul 20 11:00:23.965123 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.965087    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g" event={"ID":"b36406ce-a27c-491f-b0a3-aaa1346aba68","Type":"ContainerDied","Data":"bd38e4dc0ef167c1bec411e97fb1b5807f917208d14e175c8b99087508ea0f95"}
Jul 20 11:00:23.965655 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.965172    2561 scope.go:117] "RemoveContainer" containerID="b3be490932f844567cd7da1c4837bdb635bfb6d48be59b8895e45820b540ae84"
Jul 20 11:00:23.973766 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.973749    2561 scope.go:117] "RemoveContainer" containerID="3f04b22847366b3974ed26899d01467904673043eae6f9eba972e0ac5474e21c"
Jul 20 11:00:23.980843 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.980828    2561 scope.go:117] "RemoveContainer" containerID="42aecee09052c0a188fb42cb2fe81ffa521e57dcc534e248dd371664d7fefd66"
Jul 20 11:00:23.988616 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.988594    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"]
Jul 20 11:00:23.994663 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:23.994644    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fs54g"]
Jul 20 11:00:24.765464 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:24.765432    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" path="/var/lib/kubelet/pods/b36406ce-a27c-491f-b0a3-aaa1346aba68/volumes"
Jul 20 11:00:25.975083 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:25.975002    2561 generic.go:356] "Generic (PLEG): container finished" podID="a338166e-9993-4092-a178-98130d596df8" containerID="e7d8c4f2a313caad710391ddd705857f5befdd3fd8aecb163dfde9f220709acc" exitCode=0
Jul 20 11:00:25.975465 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:25.975075    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" event={"ID":"a338166e-9993-4092-a178-98130d596df8","Type":"ContainerDied","Data":"e7d8c4f2a313caad710391ddd705857f5befdd3fd8aecb163dfde9f220709acc"}
Jul 20 11:00:26.980663 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:26.980629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" event={"ID":"a338166e-9993-4092-a178-98130d596df8","Type":"ContainerStarted","Data":"8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911"}
Jul 20 11:00:26.981018 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:26.980692    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" event={"ID":"a338166e-9993-4092-a178-98130d596df8","Type":"ContainerStarted","Data":"58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef"}
Jul 20 11:00:26.981018 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:26.980887    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:27.000551 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:27.000509    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podStartSLOduration=2.323274141 podStartE2EDuration="6.000494489s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.677220348s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:00:26.999361588 +0000 UTC m=+3806.837460214" watchObservedRunningTime="2026-07-20 11:00:27.000494489 +0000 UTC m=+3806.838593076"
Jul 20 11:00:27.984603 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:27.984571    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:27.985870 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:27.985843    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Jul 20 11:00:28.988522 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:28.988477    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Jul 20 11:00:33.993474 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:33.993448    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:00:33.993969 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:33.993942    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Jul 20 11:00:43.994772 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:43.994733    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Jul 20 11:00:53.994330 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:00:53.994291    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Jul 20 11:01:03.994324 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:03.994286    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Jul 20 11:01:13.994341 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:13.994299    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Jul 20 11:01:23.994328 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:23.994250    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Jul 20 11:01:33.994738 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:33.994699    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Jul 20 11:01:43.994758 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:43.994728    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:01:51.786818 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:51.786783    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"]
Jul 20 11:01:51.787275 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:51.787046    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container" containerID="cri-o://58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef" gracePeriod=30
Jul 20 11:01:51.787275 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:51.787076    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kube-rbac-proxy" containerID="cri-o://8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911" gracePeriod=30
Jul 20 11:01:52.288016 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:52.287982    2561 generic.go:356] "Generic (PLEG): container finished" podID="a338166e-9993-4092-a178-98130d596df8" containerID="8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911" exitCode=2
Jul 20 11:01:52.288207 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:52.288052    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" event={"ID":"a338166e-9993-4092-a178-98130d596df8","Type":"ContainerDied","Data":"8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911"}
Jul 20 11:01:53.989015 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:53.988974    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Jul 20 11:01:53.994585 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:53.994554    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Jul 20 11:01:55.928401 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:55.928379    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:01:56.025862 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.025787    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a338166e-9993-4092-a178-98130d596df8-cabundle-cert\") pod \"a338166e-9993-4092-a178-98130d596df8\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") "
Jul 20 11:01:56.026000 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.025871    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9rc9\" (UniqueName: \"kubernetes.io/projected/a338166e-9993-4092-a178-98130d596df8-kube-api-access-h9rc9\") pod \"a338166e-9993-4092-a178-98130d596df8\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") "
Jul 20 11:01:56.026000 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.025896    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a338166e-9993-4092-a178-98130d596df8-kube-rbac-proxy-sar-config\") pod \"a338166e-9993-4092-a178-98130d596df8\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") "
Jul 20 11:01:56.026000 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.025930    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a338166e-9993-4092-a178-98130d596df8-proxy-tls\") pod \"a338166e-9993-4092-a178-98130d596df8\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") "
Jul 20 11:01:56.026000 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.025970    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a338166e-9993-4092-a178-98130d596df8-kserve-provision-location\") pod \"a338166e-9993-4092-a178-98130d596df8\" (UID: \"a338166e-9993-4092-a178-98130d596df8\") "
Jul 20 11:01:56.026242 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.026163    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a338166e-9993-4092-a178-98130d596df8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a338166e-9993-4092-a178-98130d596df8" (UID: "a338166e-9993-4092-a178-98130d596df8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:01:56.026325 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.026301    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a338166e-9993-4092-a178-98130d596df8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a338166e-9993-4092-a178-98130d596df8" (UID: "a338166e-9993-4092-a178-98130d596df8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:01:56.026370 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.026347    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a338166e-9993-4092-a178-98130d596df8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a338166e-9993-4092-a178-98130d596df8" (UID: "a338166e-9993-4092-a178-98130d596df8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:01:56.027923 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.027896    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a338166e-9993-4092-a178-98130d596df8-kube-api-access-h9rc9" (OuterVolumeSpecName: "kube-api-access-h9rc9") pod "a338166e-9993-4092-a178-98130d596df8" (UID: "a338166e-9993-4092-a178-98130d596df8"). InnerVolumeSpecName "kube-api-access-h9rc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:01:56.028030 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.027938    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a338166e-9993-4092-a178-98130d596df8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a338166e-9993-4092-a178-98130d596df8" (UID: "a338166e-9993-4092-a178-98130d596df8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:01:56.126685 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.126659    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9rc9\" (UniqueName: \"kubernetes.io/projected/a338166e-9993-4092-a178-98130d596df8-kube-api-access-h9rc9\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:01:56.126685 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.126679    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a338166e-9993-4092-a178-98130d596df8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:01:56.126825 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.126707    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a338166e-9993-4092-a178-98130d596df8-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:01:56.126825 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.126717    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a338166e-9993-4092-a178-98130d596df8-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:01:56.126825 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.126725    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a338166e-9993-4092-a178-98130d596df8-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:01:56.304317 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.304232    2561 generic.go:356] "Generic (PLEG): container finished" podID="a338166e-9993-4092-a178-98130d596df8" containerID="58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef" exitCode=0
Jul 20 11:01:56.304317 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.304294    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"
Jul 20 11:01:56.304527 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.304297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" event={"ID":"a338166e-9993-4092-a178-98130d596df8","Type":"ContainerDied","Data":"58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef"}
Jul 20 11:01:56.304527 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.304419    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl" event={"ID":"a338166e-9993-4092-a178-98130d596df8","Type":"ContainerDied","Data":"8cb203182bf0481ce206d9f0ba8a12f56e8a44ffa8378c6d85bf2863427ebc91"}
Jul 20 11:01:56.304527 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.304448    2561 scope.go:117] "RemoveContainer" containerID="8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911"
Jul 20 11:01:56.313106 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.313089    2561 scope.go:117] "RemoveContainer" containerID="58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef"
Jul 20 11:01:56.320193 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.320177    2561 scope.go:117] "RemoveContainer" containerID="e7d8c4f2a313caad710391ddd705857f5befdd3fd8aecb163dfde9f220709acc"
Jul 20 11:01:56.327255 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.327233    2561 scope.go:117] "RemoveContainer" containerID="8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911"
Jul 20 11:01:56.327528 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:01:56.327509    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911\": container with ID starting with 8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911 not found: ID does not exist" containerID="8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911"
Jul 20 11:01:56.327573 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.327537    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911"} err="failed to get container status \"8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911\": rpc error: code = NotFound desc = could not find container \"8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911\": container with ID starting with 8517d95902cf1eb31683b62f13efaf0dc909613a21216700e89629daa7897911 not found: ID does not exist"
Jul 20 11:01:56.327573 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.327554    2561 scope.go:117] "RemoveContainer" containerID="58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef"
Jul 20 11:01:56.327660 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.327577    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"]
Jul 20 11:01:56.327779 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:01:56.327762    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef\": container with ID starting with 58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef not found: ID does not exist" containerID="58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef"
Jul 20 11:01:56.327814 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.327785    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef"} err="failed to get container status \"58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef\": rpc error: code = NotFound desc = could not find container \"58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef\": container with ID starting with 58642f8d46d3e3f8b291b191e9ba1d9e0060de64d0daeaf98dd9739061ed7eef not found: ID does not exist"
Jul 20 11:01:56.327814 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.327801    2561 scope.go:117] "RemoveContainer" containerID="e7d8c4f2a313caad710391ddd705857f5befdd3fd8aecb163dfde9f220709acc"
Jul 20 11:01:56.328036 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:01:56.328020    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d8c4f2a313caad710391ddd705857f5befdd3fd8aecb163dfde9f220709acc\": container with ID starting with e7d8c4f2a313caad710391ddd705857f5befdd3fd8aecb163dfde9f220709acc not found: ID does not exist" containerID="e7d8c4f2a313caad710391ddd705857f5befdd3fd8aecb163dfde9f220709acc"
Jul 20 11:01:56.328099 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.328041    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d8c4f2a313caad710391ddd705857f5befdd3fd8aecb163dfde9f220709acc"} err="failed to get container status \"e7d8c4f2a313caad710391ddd705857f5befdd3fd8aecb163dfde9f220709acc\": rpc error: code = NotFound desc = could not find container \"e7d8c4f2a313caad710391ddd705857f5befdd3fd8aecb163dfde9f220709acc\": container with ID starting with e7d8c4f2a313caad710391ddd705857f5befdd3fd8aecb163dfde9f220709acc not found: ID does not exist"
Jul 20 11:01:56.332586 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.332565    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b7bddf5b4-8pdxl"]
Jul 20 11:01:56.769698 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:56.769670    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a338166e-9993-4092-a178-98130d596df8" path="/var/lib/kubelet/pods/a338166e-9993-4092-a178-98130d596df8/volumes"
Jul 20 11:01:57.001492 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001462    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"]
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001712    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerName="kserve-container"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001724    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerName="kserve-container"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001733    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001738    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001748    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerName="storage-initializer"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001754    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerName="storage-initializer"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001760    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kube-rbac-proxy"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001766    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kube-rbac-proxy"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001777    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerName="kube-rbac-proxy"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001782    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerName="kube-rbac-proxy"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001788    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="storage-initializer"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001793    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="storage-initializer"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001832    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kserve-container"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001839    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerName="kserve-container"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001844    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a338166e-9993-4092-a178-98130d596df8" containerName="kube-rbac-proxy"
Jul 20 11:01:57.001844 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.001850    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36406ce-a27c-491f-b0a3-aaa1346aba68" containerName="kube-rbac-proxy"
Jul 20 11:01:57.006054 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.006036    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.008945 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.008926    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 11:01:57.009143 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.009130    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 11:01:57.010149 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.010107    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 20 11:01:57.010267 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.010102    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 11:01:57.010267 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.010166    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 20 11:01:57.010267 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.010131    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 11:01:57.010403 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.010299    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 11:01:57.015918 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.015892    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"]
Jul 20 11:01:57.134687 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.134660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201927a7-35ac-4159-8574-b3b0037ac20c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.134847 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.134694    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201927a7-35ac-4159-8574-b3b0037ac20c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.134847 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.134721    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201927a7-35ac-4159-8574-b3b0037ac20c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.134847 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.134749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rn46\" (UniqueName: \"kubernetes.io/projected/201927a7-35ac-4159-8574-b3b0037ac20c-kube-api-access-2rn46\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.134847 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.134807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201927a7-35ac-4159-8574-b3b0037ac20c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.235715 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.235671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201927a7-35ac-4159-8574-b3b0037ac20c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.235911 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.235734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201927a7-35ac-4159-8574-b3b0037ac20c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.235911 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.235761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201927a7-35ac-4159-8574-b3b0037ac20c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.235911 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.235795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201927a7-35ac-4159-8574-b3b0037ac20c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.236087 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.235935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rn46\" (UniqueName: \"kubernetes.io/projected/201927a7-35ac-4159-8574-b3b0037ac20c-kube-api-access-2rn46\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.236191 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.236170    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201927a7-35ac-4159-8574-b3b0037ac20c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.236530 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.236509    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201927a7-35ac-4159-8574-b3b0037ac20c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.236530 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.236526    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201927a7-35ac-4159-8574-b3b0037ac20c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.238256 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.238240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201927a7-35ac-4159-8574-b3b0037ac20c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.245153 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.245100    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rn46\" (UniqueName: \"kubernetes.io/projected/201927a7-35ac-4159-8574-b3b0037ac20c-kube-api-access-2rn46\") pod \"isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.317534 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.317509    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:01:57.442891 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:57.442865    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"]
Jul 20 11:01:57.445415 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:01:57.445390    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201927a7_35ac_4159_8574_b3b0037ac20c.slice/crio-2189826b27842ae908745cab08da12f84cff321c71c84177f25c44e2c6e8cd12 WatchSource:0}: Error finding container 2189826b27842ae908745cab08da12f84cff321c71c84177f25c44e2c6e8cd12: Status 404 returned error can't find the container with id 2189826b27842ae908745cab08da12f84cff321c71c84177f25c44e2c6e8cd12
Jul 20 11:01:58.313496 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:58.313459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" event={"ID":"201927a7-35ac-4159-8574-b3b0037ac20c","Type":"ContainerStarted","Data":"1e8ced98d15d6c81d8dbbf43ea5c120407e6f0815c2a80f58750f1c318bd9143"}
Jul 20 11:01:58.313916 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:01:58.313518    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" event={"ID":"201927a7-35ac-4159-8574-b3b0037ac20c","Type":"ContainerStarted","Data":"2189826b27842ae908745cab08da12f84cff321c71c84177f25c44e2c6e8cd12"}
Jul 20 11:02:01.324992 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:01.324958    2561 generic.go:356] "Generic (PLEG): container finished" podID="201927a7-35ac-4159-8574-b3b0037ac20c" containerID="1e8ced98d15d6c81d8dbbf43ea5c120407e6f0815c2a80f58750f1c318bd9143" exitCode=0
Jul 20 11:02:01.325389 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:01.325036    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" event={"ID":"201927a7-35ac-4159-8574-b3b0037ac20c","Type":"ContainerDied","Data":"1e8ced98d15d6c81d8dbbf43ea5c120407e6f0815c2a80f58750f1c318bd9143"}
Jul 20 11:02:02.330785 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:02.330751    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" event={"ID":"201927a7-35ac-4159-8574-b3b0037ac20c","Type":"ContainerStarted","Data":"b3569a924b7fc072b148a7801e262f2840bafbc3a76292a24e0a7010b426413e"}
Jul 20 11:02:02.331183 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:02.330835    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" event={"ID":"201927a7-35ac-4159-8574-b3b0037ac20c","Type":"ContainerStarted","Data":"d18f96861caad6806630c179a8bf4d7945dcb16dda3ae6e55fa4b4fe44860716"}
Jul 20 11:02:02.331183 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:02.331140    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:02:02.331300 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:02.331282    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:02:02.332576 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:02.332553    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 20 11:02:02.354337 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:02.354295    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podStartSLOduration=2.655019285 podStartE2EDuration="6.354280707s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.699261422s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:02:02.352911778 +0000 UTC m=+3902.191010362" watchObservedRunningTime="2026-07-20 11:02:02.354280707 +0000 UTC m=+3902.192379293"
Jul 20 11:02:03.334649 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:03.334608    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 20 11:02:04.337793 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:04.337755    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 20 11:02:09.342759 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:09.342730    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:02:09.343277 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:09.343251    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 20 11:02:16.206717 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:16.206690    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 11:02:16.214323 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:16.214302    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 11:02:19.343305 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:19.343270    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 20 11:02:29.343475 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:29.343423    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 20 11:02:39.343324 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:39.343283    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 20 11:02:49.343271 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:49.343230    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 20 11:02:59.343347 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:02:59.343308    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 20 11:03:09.343576 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:09.343538    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 20 11:03:19.344246 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:19.344220    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:03:27.139358 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:27.139323    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"]
Jul 20 11:03:27.139859 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:27.139565    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container" containerID="cri-o://d18f96861caad6806630c179a8bf4d7945dcb16dda3ae6e55fa4b4fe44860716" gracePeriod=30
Jul 20 11:03:27.139859 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:27.139651    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kube-rbac-proxy" containerID="cri-o://b3569a924b7fc072b148a7801e262f2840bafbc3a76292a24e0a7010b426413e" gracePeriod=30
Jul 20 11:03:27.639635 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:27.639604    2561 generic.go:356] "Generic (PLEG): container finished" podID="201927a7-35ac-4159-8574-b3b0037ac20c" containerID="b3569a924b7fc072b148a7801e262f2840bafbc3a76292a24e0a7010b426413e" exitCode=2
Jul 20 11:03:27.639797 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:27.639675    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" event={"ID":"201927a7-35ac-4159-8574-b3b0037ac20c","Type":"ContainerDied","Data":"b3569a924b7fc072b148a7801e262f2840bafbc3a76292a24e0a7010b426413e"}
Jul 20 11:03:29.338240 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:29.338201    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Jul 20 11:03:29.343583 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:29.343559    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 20 11:03:31.656866 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.656827    2561 generic.go:356] "Generic (PLEG): container finished" podID="201927a7-35ac-4159-8574-b3b0037ac20c" containerID="d18f96861caad6806630c179a8bf4d7945dcb16dda3ae6e55fa4b4fe44860716" exitCode=0
Jul 20 11:03:31.657277 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.656866    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" event={"ID":"201927a7-35ac-4159-8574-b3b0037ac20c","Type":"ContainerDied","Data":"d18f96861caad6806630c179a8bf4d7945dcb16dda3ae6e55fa4b4fe44860716"}
Jul 20 11:03:31.691046 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.691026    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:03:31.798826 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.798741    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201927a7-35ac-4159-8574-b3b0037ac20c-cabundle-cert\") pod \"201927a7-35ac-4159-8574-b3b0037ac20c\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") "
Jul 20 11:03:31.798826 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.798792    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201927a7-35ac-4159-8574-b3b0037ac20c-kserve-provision-location\") pod \"201927a7-35ac-4159-8574-b3b0037ac20c\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") "
Jul 20 11:03:31.799033 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.798898    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201927a7-35ac-4159-8574-b3b0037ac20c-kube-rbac-proxy-sar-config\") pod \"201927a7-35ac-4159-8574-b3b0037ac20c\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") "
Jul 20 11:03:31.799033 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.798955    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201927a7-35ac-4159-8574-b3b0037ac20c-proxy-tls\") pod \"201927a7-35ac-4159-8574-b3b0037ac20c\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") "
Jul 20 11:03:31.799033 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.799017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rn46\" (UniqueName: \"kubernetes.io/projected/201927a7-35ac-4159-8574-b3b0037ac20c-kube-api-access-2rn46\") pod \"201927a7-35ac-4159-8574-b3b0037ac20c\" (UID: \"201927a7-35ac-4159-8574-b3b0037ac20c\") "
Jul 20 11:03:31.799229 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.799061    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201927a7-35ac-4159-8574-b3b0037ac20c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "201927a7-35ac-4159-8574-b3b0037ac20c" (UID: "201927a7-35ac-4159-8574-b3b0037ac20c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:03:31.799229 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.799184    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201927a7-35ac-4159-8574-b3b0037ac20c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "201927a7-35ac-4159-8574-b3b0037ac20c" (UID: "201927a7-35ac-4159-8574-b3b0037ac20c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:03:31.799339 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.799243    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201927a7-35ac-4159-8574-b3b0037ac20c-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:03:31.799339 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.799245    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201927a7-35ac-4159-8574-b3b0037ac20c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "201927a7-35ac-4159-8574-b3b0037ac20c" (UID: "201927a7-35ac-4159-8574-b3b0037ac20c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:03:31.799339 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.799256    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201927a7-35ac-4159-8574-b3b0037ac20c-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:03:31.801020 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.800994    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201927a7-35ac-4159-8574-b3b0037ac20c-kube-api-access-2rn46" (OuterVolumeSpecName: "kube-api-access-2rn46") pod "201927a7-35ac-4159-8574-b3b0037ac20c" (UID: "201927a7-35ac-4159-8574-b3b0037ac20c"). InnerVolumeSpecName "kube-api-access-2rn46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:03:31.801101 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.801026    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201927a7-35ac-4159-8574-b3b0037ac20c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "201927a7-35ac-4159-8574-b3b0037ac20c" (UID: "201927a7-35ac-4159-8574-b3b0037ac20c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:03:31.900253 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.900204    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201927a7-35ac-4159-8574-b3b0037ac20c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:03:31.900253 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.900247    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201927a7-35ac-4159-8574-b3b0037ac20c-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:03:31.900253 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:31.900260    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rn46\" (UniqueName: \"kubernetes.io/projected/201927a7-35ac-4159-8574-b3b0037ac20c-kube-api-access-2rn46\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:03:32.291401 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.291363    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"]
Jul 20 11:03:32.291637 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.291625    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kube-rbac-proxy"
Jul 20 11:03:32.291682 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.291638    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kube-rbac-proxy"
Jul 20 11:03:32.291682 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.291648    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container"
Jul 20 11:03:32.291682 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.291653    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container"
Jul 20 11:03:32.291682 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.291663    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="storage-initializer"
Jul 20 11:03:32.291682 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.291668    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="storage-initializer"
Jul 20 11:03:32.291863 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.291712    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kube-rbac-proxy"
Jul 20 11:03:32.291863 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.291720    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" containerName="kserve-container"
Jul 20 11:03:32.295004 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.294982    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.297513 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.297487    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 20 11:03:32.297622 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.297512    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 20 11:03:32.305921 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.305894    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"]
Jul 20 11:03:32.405228 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.405192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrbmt\" (UniqueName: \"kubernetes.io/projected/44a41ba3-edf0-42f9-91e5-667211114a77-kube-api-access-lrbmt\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.405228 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.405232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a41ba3-edf0-42f9-91e5-667211114a77-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.405423 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.405320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a41ba3-edf0-42f9-91e5-667211114a77-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.405423 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.405355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44a41ba3-edf0-42f9-91e5-667211114a77-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.405423 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.405375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44a41ba3-edf0-42f9-91e5-667211114a77-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.505920 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.505884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrbmt\" (UniqueName: \"kubernetes.io/projected/44a41ba3-edf0-42f9-91e5-667211114a77-kube-api-access-lrbmt\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.506067 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.505930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a41ba3-edf0-42f9-91e5-667211114a77-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.506067 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.505983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a41ba3-edf0-42f9-91e5-667211114a77-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.506067 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.506010    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44a41ba3-edf0-42f9-91e5-667211114a77-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.506263 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:03:32.506090    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Jul 20 11:03:32.506263 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:03:32.506169    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a41ba3-edf0-42f9-91e5-667211114a77-proxy-tls podName:44a41ba3-edf0-42f9-91e5-667211114a77 nodeName:}" failed. No retries permitted until 2026-07-20 11:03:33.006148675 +0000 UTC m=+3992.844247238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/44a41ba3-edf0-42f9-91e5-667211114a77-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-d6ppk" (UID: "44a41ba3-edf0-42f9-91e5-667211114a77") : secret "isvc-tensorflow-predictor-serving-cert" not found
Jul 20 11:03:32.506263 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.506162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44a41ba3-edf0-42f9-91e5-667211114a77-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.506524 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.506502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44a41ba3-edf0-42f9-91e5-667211114a77-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.506643 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.506622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a41ba3-edf0-42f9-91e5-667211114a77-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.506785 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.506766    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44a41ba3-edf0-42f9-91e5-667211114a77-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.515425 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.515398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrbmt\" (UniqueName: \"kubernetes.io/projected/44a41ba3-edf0-42f9-91e5-667211114a77-kube-api-access-lrbmt\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:32.663393 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.663367    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"
Jul 20 11:03:32.663788 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.663362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl" event={"ID":"201927a7-35ac-4159-8574-b3b0037ac20c","Type":"ContainerDied","Data":"2189826b27842ae908745cab08da12f84cff321c71c84177f25c44e2c6e8cd12"}
Jul 20 11:03:32.663788 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.663500    2561 scope.go:117] "RemoveContainer" containerID="b3569a924b7fc072b148a7801e262f2840bafbc3a76292a24e0a7010b426413e"
Jul 20 11:03:32.672590 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.672570    2561 scope.go:117] "RemoveContainer" containerID="d18f96861caad6806630c179a8bf4d7945dcb16dda3ae6e55fa4b4fe44860716"
Jul 20 11:03:32.684727 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.684702    2561 scope.go:117] "RemoveContainer" containerID="1e8ced98d15d6c81d8dbbf43ea5c120407e6f0815c2a80f58750f1c318bd9143"
Jul 20 11:03:32.686031 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.686011    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"]
Jul 20 11:03:32.691302 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.691281    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64449768df-tc4xl"]
Jul 20 11:03:32.764795 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:32.764765    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="201927a7-35ac-4159-8574-b3b0037ac20c" path="/var/lib/kubelet/pods/201927a7-35ac-4159-8574-b3b0037ac20c/volumes"
Jul 20 11:03:33.010349 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:33.010266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a41ba3-edf0-42f9-91e5-667211114a77-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:33.012592 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:33.012574    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a41ba3-edf0-42f9-91e5-667211114a77-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-d6ppk\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:33.206033 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:33.205999    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:33.335037 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:33.334991    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"]
Jul 20 11:03:33.338081 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:03:33.338052    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a41ba3_edf0_42f9_91e5_667211114a77.slice/crio-e190f9641de953e2a81474567e888665efbfe2b813a50bc883a3922fe5c00ac4 WatchSource:0}: Error finding container e190f9641de953e2a81474567e888665efbfe2b813a50bc883a3922fe5c00ac4: Status 404 returned error can't find the container with id e190f9641de953e2a81474567e888665efbfe2b813a50bc883a3922fe5c00ac4
Jul 20 11:03:33.669998 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:33.669965    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" event={"ID":"44a41ba3-edf0-42f9-91e5-667211114a77","Type":"ContainerStarted","Data":"6ee62db05fe807e3bfe98a5ac112492cbf24e391e1a7c8a97fb8af06beacbc61"}
Jul 20 11:03:33.670414 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:33.670018    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" event={"ID":"44a41ba3-edf0-42f9-91e5-667211114a77","Type":"ContainerStarted","Data":"e190f9641de953e2a81474567e888665efbfe2b813a50bc883a3922fe5c00ac4"}
Jul 20 11:03:37.689085 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:37.689047    2561 generic.go:356] "Generic (PLEG): container finished" podID="44a41ba3-edf0-42f9-91e5-667211114a77" containerID="6ee62db05fe807e3bfe98a5ac112492cbf24e391e1a7c8a97fb8af06beacbc61" exitCode=0
Jul 20 11:03:37.689488 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:37.689097    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" event={"ID":"44a41ba3-edf0-42f9-91e5-667211114a77","Type":"ContainerDied","Data":"6ee62db05fe807e3bfe98a5ac112492cbf24e391e1a7c8a97fb8af06beacbc61"}
Jul 20 11:03:37.690286 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:37.690270    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 11:03:41.707350 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:41.707320    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" event={"ID":"44a41ba3-edf0-42f9-91e5-667211114a77","Type":"ContainerStarted","Data":"2303c12d79c866e5514c00b445bbf3c07bf45a4ab304b700ea62303dd3999e65"}
Jul 20 11:03:41.707625 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:41.707410    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" event={"ID":"44a41ba3-edf0-42f9-91e5-667211114a77","Type":"ContainerStarted","Data":"55b96662731edc6944e2a21d9efb22f42126320b3c4757acd16f67ea29153c83"}
Jul 20 11:03:41.707625 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:41.707602    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:41.728441 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:41.728398    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" podStartSLOduration=1.694704727 podStartE2EDuration="9.728384776s" totalImagesPullingTime="3.867917125s" totalInitContainerRuntime="4.165762924s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:03:41.726546759 +0000 UTC m=+4001.564645362" watchObservedRunningTime="2026-07-20 11:03:41.728384776 +0000 UTC m=+4001.566483361"
Jul 20 11:03:42.711612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:42.711579    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:42.712953 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:42.712925    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Jul 20 11:03:43.715656 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:43.715616    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Jul 20 11:03:48.720055 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:48.720027    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:03:48.720618 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:48.720591    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Jul 20 11:03:58.721692 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:03:58.721660    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:04:12.410646 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:12.410611    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"]
Jul 20 11:04:12.411164 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:12.410927    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kserve-container" containerID="cri-o://55b96662731edc6944e2a21d9efb22f42126320b3c4757acd16f67ea29153c83" gracePeriod=30
Jul 20 11:04:12.411267 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:12.411230    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kube-rbac-proxy" containerID="cri-o://2303c12d79c866e5514c00b445bbf3c07bf45a4ab304b700ea62303dd3999e65" gracePeriod=30
Jul 20 11:04:12.821228 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:12.821146    2561 generic.go:356] "Generic (PLEG): container finished" podID="44a41ba3-edf0-42f9-91e5-667211114a77" containerID="2303c12d79c866e5514c00b445bbf3c07bf45a4ab304b700ea62303dd3999e65" exitCode=2
Jul 20 11:04:12.821228 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:12.821182    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" event={"ID":"44a41ba3-edf0-42f9-91e5-667211114a77","Type":"ContainerDied","Data":"2303c12d79c866e5514c00b445bbf3c07bf45a4ab304b700ea62303dd3999e65"}
Jul 20 11:04:13.716048 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:13.716008    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Jul 20 11:04:17.573661 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.573622    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"]
Jul 20 11:04:17.576921 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.576900    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.579500 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.579478    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 20 11:04:17.579500 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.579490    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 20 11:04:17.587761 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.587741    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"]
Jul 20 11:04:17.636816 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.636785    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.636933 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.636850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vzql\" (UniqueName: \"kubernetes.io/projected/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kube-api-access-7vzql\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.636933 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.636882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.636933 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.636923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.637047 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.636941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.738020 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.737989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.738201 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.738047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vzql\" (UniqueName: \"kubernetes.io/projected/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kube-api-access-7vzql\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.738201 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.738071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.738201 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.738090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.738201 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.738134    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.738620 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.738595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.738911 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.738894    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.738964 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.738891    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.740550 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.740519    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.746589 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.746568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vzql\" (UniqueName: \"kubernetes.io/projected/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kube-api-access-7vzql\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-29kgh\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:17.887741 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:17.887713    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:18.017934 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:18.017908    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"]
Jul 20 11:04:18.020343 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:04:18.020312    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c1bbd4_d6cd_4ac8_8321_ab25a8a98cc7.slice/crio-b079aec9577ceff78d5db2e66019bcbbbec361a66bca288a2a82fa69785b61e5 WatchSource:0}: Error finding container b079aec9577ceff78d5db2e66019bcbbbec361a66bca288a2a82fa69785b61e5: Status 404 returned error can't find the container with id b079aec9577ceff78d5db2e66019bcbbbec361a66bca288a2a82fa69785b61e5
Jul 20 11:04:18.716421 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:18.716375    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Jul 20 11:04:18.842374 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:18.842344    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" event={"ID":"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7","Type":"ContainerStarted","Data":"175e3ac47f2e4b6999dc2f6806c114c3954e5468769e55ae436cfadc5a3839cc"}
Jul 20 11:04:18.842511 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:18.842398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" event={"ID":"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7","Type":"ContainerStarted","Data":"b079aec9577ceff78d5db2e66019bcbbbec361a66bca288a2a82fa69785b61e5"}
Jul 20 11:04:23.716497 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:23.716455    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Jul 20 11:04:23.716877 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:23.716590    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:04:23.862346 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:23.862311    2561 generic.go:356] "Generic (PLEG): container finished" podID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerID="175e3ac47f2e4b6999dc2f6806c114c3954e5468769e55ae436cfadc5a3839cc" exitCode=0
Jul 20 11:04:23.862346 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:23.862349    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" event={"ID":"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7","Type":"ContainerDied","Data":"175e3ac47f2e4b6999dc2f6806c114c3954e5468769e55ae436cfadc5a3839cc"}
Jul 20 11:04:24.868454 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:24.868418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" event={"ID":"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7","Type":"ContainerStarted","Data":"9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996"}
Jul 20 11:04:24.868940 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:24.868489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" event={"ID":"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7","Type":"ContainerStarted","Data":"a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4"}
Jul 20 11:04:24.868940 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:24.868790    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:24.888605 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:24.888559    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" podStartSLOduration=2.685709322 podStartE2EDuration="7.888544868s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.202835546s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:04:24.887580456 +0000 UTC m=+4044.725679073" watchObservedRunningTime="2026-07-20 11:04:24.888544868 +0000 UTC m=+4044.726643436"
Jul 20 11:04:25.872428 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:25.872397    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:25.873593 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:25.873563    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Jul 20 11:04:26.875623 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:26.875582    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Jul 20 11:04:28.716127 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:28.716074    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Jul 20 11:04:31.879657 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:31.879627    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:31.880309 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:31.880280    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Jul 20 11:04:33.716795 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:33.716752    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Jul 20 11:04:38.716364 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:38.716318    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Jul 20 11:04:41.880688 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:41.880659    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:04:42.941362 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:42.941334    2561 generic.go:356] "Generic (PLEG): container finished" podID="44a41ba3-edf0-42f9-91e5-667211114a77" containerID="55b96662731edc6944e2a21d9efb22f42126320b3c4757acd16f67ea29153c83" exitCode=137
Jul 20 11:04:42.941712 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:42.941409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" event={"ID":"44a41ba3-edf0-42f9-91e5-667211114a77","Type":"ContainerDied","Data":"55b96662731edc6944e2a21d9efb22f42126320b3c4757acd16f67ea29153c83"}
Jul 20 11:04:43.055870 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.055848    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:04:43.137745 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.137714    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a41ba3-edf0-42f9-91e5-667211114a77-kube-rbac-proxy-sar-config\") pod \"44a41ba3-edf0-42f9-91e5-667211114a77\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") "
Jul 20 11:04:43.137918 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.137751    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44a41ba3-edf0-42f9-91e5-667211114a77-kserve-provision-location\") pod \"44a41ba3-edf0-42f9-91e5-667211114a77\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") "
Jul 20 11:04:43.137918 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.137823    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44a41ba3-edf0-42f9-91e5-667211114a77-cabundle-cert\") pod \"44a41ba3-edf0-42f9-91e5-667211114a77\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") "
Jul 20 11:04:43.137918 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.137856    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrbmt\" (UniqueName: \"kubernetes.io/projected/44a41ba3-edf0-42f9-91e5-667211114a77-kube-api-access-lrbmt\") pod \"44a41ba3-edf0-42f9-91e5-667211114a77\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") "
Jul 20 11:04:43.137918 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.137879    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a41ba3-edf0-42f9-91e5-667211114a77-proxy-tls\") pod \"44a41ba3-edf0-42f9-91e5-667211114a77\" (UID: \"44a41ba3-edf0-42f9-91e5-667211114a77\") "
Jul 20 11:04:43.138261 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.138236    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a41ba3-edf0-42f9-91e5-667211114a77-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44a41ba3-edf0-42f9-91e5-667211114a77" (UID: "44a41ba3-edf0-42f9-91e5-667211114a77"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:04:43.138325 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.138239    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a41ba3-edf0-42f9-91e5-667211114a77-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "44a41ba3-edf0-42f9-91e5-667211114a77" (UID: "44a41ba3-edf0-42f9-91e5-667211114a77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:04:43.139952 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.139931    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44a41ba3-edf0-42f9-91e5-667211114a77-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44a41ba3-edf0-42f9-91e5-667211114a77" (UID: "44a41ba3-edf0-42f9-91e5-667211114a77"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:04:43.140095 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.140066    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a41ba3-edf0-42f9-91e5-667211114a77-kube-api-access-lrbmt" (OuterVolumeSpecName: "kube-api-access-lrbmt") pod "44a41ba3-edf0-42f9-91e5-667211114a77" (UID: "44a41ba3-edf0-42f9-91e5-667211114a77"). InnerVolumeSpecName "kube-api-access-lrbmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:04:43.148622 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.148599    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a41ba3-edf0-42f9-91e5-667211114a77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44a41ba3-edf0-42f9-91e5-667211114a77" (UID: "44a41ba3-edf0-42f9-91e5-667211114a77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:04:43.238583 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.238505    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a41ba3-edf0-42f9-91e5-667211114a77-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:04:43.238583 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.238536    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44a41ba3-edf0-42f9-91e5-667211114a77-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:04:43.238583 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.238547    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44a41ba3-edf0-42f9-91e5-667211114a77-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:04:43.238583 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.238555    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrbmt\" (UniqueName: \"kubernetes.io/projected/44a41ba3-edf0-42f9-91e5-667211114a77-kube-api-access-lrbmt\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:04:43.238583 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.238563    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a41ba3-edf0-42f9-91e5-667211114a77-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:04:43.946727 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.946695    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"
Jul 20 11:04:43.946727 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.946706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk" event={"ID":"44a41ba3-edf0-42f9-91e5-667211114a77","Type":"ContainerDied","Data":"e190f9641de953e2a81474567e888665efbfe2b813a50bc883a3922fe5c00ac4"}
Jul 20 11:04:43.947260 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.946777    2561 scope.go:117] "RemoveContainer" containerID="2303c12d79c866e5514c00b445bbf3c07bf45a4ab304b700ea62303dd3999e65"
Jul 20 11:04:43.955138 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.955102    2561 scope.go:117] "RemoveContainer" containerID="55b96662731edc6944e2a21d9efb22f42126320b3c4757acd16f67ea29153c83"
Jul 20 11:04:43.964129 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.964093    2561 scope.go:117] "RemoveContainer" containerID="6ee62db05fe807e3bfe98a5ac112492cbf24e391e1a7c8a97fb8af06beacbc61"
Jul 20 11:04:43.968934 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.968908    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"]
Jul 20 11:04:43.972807 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:43.972787    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d6ppk"]
Jul 20 11:04:44.765369 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:44.765336    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" path="/var/lib/kubelet/pods/44a41ba3-edf0-42f9-91e5-667211114a77/volumes"
Jul 20 11:04:58.113472 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:58.113439    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"]
Jul 20 11:04:58.113896 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:58.113720    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kserve-container" containerID="cri-o://a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4" gracePeriod=30
Jul 20 11:04:58.113896 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:58.113792    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kube-rbac-proxy" containerID="cri-o://9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996" gracePeriod=30
Jul 20 11:04:59.007523 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:59.007493    2561 generic.go:356] "Generic (PLEG): container finished" podID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerID="9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996" exitCode=2
Jul 20 11:04:59.007698 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:04:59.007571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" event={"ID":"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7","Type":"ContainerDied","Data":"9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996"}
Jul 20 11:05:01.876700 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:01.876658    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Jul 20 11:05:03.272222 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.272182    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"]
Jul 20 11:05:03.272715 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.272453    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kube-rbac-proxy"
Jul 20 11:05:03.272715 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.272464    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kube-rbac-proxy"
Jul 20 11:05:03.272715 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.272473    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kserve-container"
Jul 20 11:05:03.272715 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.272478    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kserve-container"
Jul 20 11:05:03.272715 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.272487    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="storage-initializer"
Jul 20 11:05:03.272715 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.272492    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="storage-initializer"
Jul 20 11:05:03.272715 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.272538    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kserve-container"
Jul 20 11:05:03.272715 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.272544    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a41ba3-edf0-42f9-91e5-667211114a77" containerName="kube-rbac-proxy"
Jul 20 11:05:03.275450 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.275423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.278007 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.277969    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 20 11:05:03.278007 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.278001    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 20 11:05:03.285655 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.285630    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"]
Jul 20 11:05:03.382632 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.382601    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.382795 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.382636    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.382795 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.382678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drnzz\" (UniqueName: \"kubernetes.io/projected/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kube-api-access-drnzz\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.382795 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.382734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.382795 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.382770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.483901 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.483852    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.483901 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.483902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.484190 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:05:03.483997    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Jul 20 11:05:03.484190 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.484014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drnzz\" (UniqueName: \"kubernetes.io/projected/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kube-api-access-drnzz\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.484190 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:05:03.484048    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-proxy-tls podName:7af4e3fa-7494-436d-bdcd-f622a6fdd15b nodeName:}" failed. No retries permitted until 2026-07-20 11:05:03.984029433 +0000 UTC m=+4083.822127995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-proxy-tls") pod "isvc-triton-predictor-5d7db58676-jprgg" (UID: "7af4e3fa-7494-436d-bdcd-f622a6fdd15b") : secret "isvc-triton-predictor-serving-cert" not found
Jul 20 11:05:03.484190 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.484070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.484190 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.484097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.484489 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.484468    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.484579 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.484564    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.484703 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.484688    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.495685 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.495668    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drnzz\" (UniqueName: \"kubernetes.io/projected/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kube-api-access-drnzz\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.986859 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.986813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:03.989240 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:03.989217    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-jprgg\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:04.187262 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:04.187212    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:05:04.319444 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:04.319290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"]
Jul 20 11:05:04.321925 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:05:04.321898    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af4e3fa_7494_436d_bdcd_f622a6fdd15b.slice/crio-f792e3d162decc3c7a1e331432c8f0ab6e67256b80dab01ca9b4ec8ca3a44112 WatchSource:0}: Error finding container f792e3d162decc3c7a1e331432c8f0ab6e67256b80dab01ca9b4ec8ca3a44112: Status 404 returned error can't find the container with id f792e3d162decc3c7a1e331432c8f0ab6e67256b80dab01ca9b4ec8ca3a44112
Jul 20 11:05:05.034933 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:05.034888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" event={"ID":"7af4e3fa-7494-436d-bdcd-f622a6fdd15b","Type":"ContainerStarted","Data":"bfc07caa4a880470ca37267ddb8173df911ed98a730225b4369e31a1341b2315"}
Jul 20 11:05:05.035137 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:05.034996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" event={"ID":"7af4e3fa-7494-436d-bdcd-f622a6fdd15b","Type":"ContainerStarted","Data":"f792e3d162decc3c7a1e331432c8f0ab6e67256b80dab01ca9b4ec8ca3a44112"}
Jul 20 11:05:06.875692 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:06.875652    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Jul 20 11:05:09.051744 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:09.051712    2561 generic.go:356] "Generic (PLEG): container finished" podID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerID="bfc07caa4a880470ca37267ddb8173df911ed98a730225b4369e31a1341b2315" exitCode=0
Jul 20 11:05:09.052104 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:09.051787    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" event={"ID":"7af4e3fa-7494-436d-bdcd-f622a6fdd15b","Type":"ContainerDied","Data":"bfc07caa4a880470ca37267ddb8173df911ed98a730225b4369e31a1341b2315"}
Jul 20 11:05:11.876762 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:11.876696    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Jul 20 11:05:11.877259 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:11.876836    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:05:16.876678 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:16.876255    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Jul 20 11:05:21.875944 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:21.875898    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Jul 20 11:05:26.876289 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:26.876242    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Jul 20 11:05:28.832450 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:28.832419    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:05:28.898968 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:28.898877    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kube-rbac-proxy-sar-config\") pod \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") "
Jul 20 11:05:28.898968 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:28.898940    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vzql\" (UniqueName: \"kubernetes.io/projected/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kube-api-access-7vzql\") pod \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") "
Jul 20 11:05:28.899246 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:28.898978    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kserve-provision-location\") pod \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") "
Jul 20 11:05:28.899246 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:28.899021    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-proxy-tls\") pod \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") "
Jul 20 11:05:28.899246 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:28.899047    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-cabundle-cert\") pod \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\" (UID: \"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7\") "
Jul 20 11:05:28.899420 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:28.899311    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" (UID: "41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:05:28.899583 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:28.899557    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" (UID: "41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:05:28.902054 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:28.902024    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kube-api-access-7vzql" (OuterVolumeSpecName: "kube-api-access-7vzql") pod "41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" (UID: "41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7"). InnerVolumeSpecName "kube-api-access-7vzql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:05:28.902482 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:28.902460    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" (UID: "41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:05:28.904070 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:28.904046    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" (UID: "41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:05:29.000651 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.000613    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:05:29.000651 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.000647    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:05:29.000884 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.000663    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:05:29.000884 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.000677    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vzql\" (UniqueName: \"kubernetes.io/projected/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kube-api-access-7vzql\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:05:29.000884 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.000691    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:05:29.173176 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.173074    2561 generic.go:356] "Generic (PLEG): container finished" podID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerID="a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4" exitCode=137
Jul 20 11:05:29.173176 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.173158    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"
Jul 20 11:05:29.173403 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.173149    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" event={"ID":"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7","Type":"ContainerDied","Data":"a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4"}
Jul 20 11:05:29.173403 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.173312    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh" event={"ID":"41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7","Type":"ContainerDied","Data":"b079aec9577ceff78d5db2e66019bcbbbec361a66bca288a2a82fa69785b61e5"}
Jul 20 11:05:29.173403 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.173362    2561 scope.go:117] "RemoveContainer" containerID="9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996"
Jul 20 11:05:29.184543 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.184526    2561 scope.go:117] "RemoveContainer" containerID="a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4"
Jul 20 11:05:29.194602 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.194578    2561 scope.go:117] "RemoveContainer" containerID="175e3ac47f2e4b6999dc2f6806c114c3954e5468769e55ae436cfadc5a3839cc"
Jul 20 11:05:29.199751 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.199709    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"]
Jul 20 11:05:29.203819 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.203799    2561 scope.go:117] "RemoveContainer" containerID="9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996"
Jul 20 11:05:29.204088 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:05:29.204055    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996\": container with ID starting with 9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996 not found: ID does not exist" containerID="9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996"
Jul 20 11:05:29.204214 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.204093    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996"} err="failed to get container status \"9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996\": rpc error: code = NotFound desc = could not find container \"9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996\": container with ID starting with 9d2ae97eef0b78bf800868515c83dc4ecabc7da160cfe82580f56c05756a5996 not found: ID does not exist"
Jul 20 11:05:29.204214 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.204136    2561 scope.go:117] "RemoveContainer" containerID="a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4"
Jul 20 11:05:29.204446 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:05:29.204423    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4\": container with ID starting with a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4 not found: ID does not exist" containerID="a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4"
Jul 20 11:05:29.204554 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.204448    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4"} err="failed to get container status \"a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4\": rpc error: code = NotFound desc = could not find container \"a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4\": container with ID starting with a1ee09c7c096a2b1689b32cffc4e6d89fcf23e5afb889a5b0354a94c49428cc4 not found: ID does not exist"
Jul 20 11:05:29.204554 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.204463    2561 scope.go:117] "RemoveContainer" containerID="175e3ac47f2e4b6999dc2f6806c114c3954e5468769e55ae436cfadc5a3839cc"
Jul 20 11:05:29.204758 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:05:29.204733    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175e3ac47f2e4b6999dc2f6806c114c3954e5468769e55ae436cfadc5a3839cc\": container with ID starting with 175e3ac47f2e4b6999dc2f6806c114c3954e5468769e55ae436cfadc5a3839cc not found: ID does not exist" containerID="175e3ac47f2e4b6999dc2f6806c114c3954e5468769e55ae436cfadc5a3839cc"
Jul 20 11:05:29.204846 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.204765    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175e3ac47f2e4b6999dc2f6806c114c3954e5468769e55ae436cfadc5a3839cc"} err="failed to get container status \"175e3ac47f2e4b6999dc2f6806c114c3954e5468769e55ae436cfadc5a3839cc\": rpc error: code = NotFound desc = could not find container \"175e3ac47f2e4b6999dc2f6806c114c3954e5468769e55ae436cfadc5a3839cc\": container with ID starting with 175e3ac47f2e4b6999dc2f6806c114c3954e5468769e55ae436cfadc5a3839cc not found: ID does not exist"
Jul 20 11:05:29.205879 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:29.205854    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-29kgh"]
Jul 20 11:05:30.766936 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:30.766898    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" path="/var/lib/kubelet/pods/41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7/volumes"
Jul 20 11:05:39.224858 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:39.224817    2561 generic.go:356] "Generic (PLEG): container finished" podID="7d5831d2-0e10-454b-9fb5-fb909c11e317" containerID="93b6449580d39cb45b8ab33bf09e2232b77b8fa69ad36ffe64a81ea0075351a1" exitCode=1
Jul 20 11:05:39.225403 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:39.224908    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-qjmq5" event={"ID":"7d5831d2-0e10-454b-9fb5-fb909c11e317","Type":"ContainerDied","Data":"93b6449580d39cb45b8ab33bf09e2232b77b8fa69ad36ffe64a81ea0075351a1"}
Jul 20 11:05:39.225403 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:39.225274    2561 scope.go:117] "RemoveContainer" containerID="93b6449580d39cb45b8ab33bf09e2232b77b8fa69ad36ffe64a81ea0075351a1"
Jul 20 11:05:41.237105 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:41.236957    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-qjmq5" event={"ID":"7d5831d2-0e10-454b-9fb5-fb909c11e317","Type":"ContainerStarted","Data":"da632d664e8edf23e9164e2f8d8c46710c9c9e03400a36c0fc1bb80ec24b0ba6"}
Jul 20 11:05:41.237690 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:41.237450    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 11:05:52.244801 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:05:52.244766    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-qjmq5"
Jul 20 11:07:04.599700 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:04.599662    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" event={"ID":"7af4e3fa-7494-436d-bdcd-f622a6fdd15b","Type":"ContainerStarted","Data":"65613bec1431463af64fdf84f9a8a3e1f540053b1c52c93ef0818e7321f954d3"}
Jul 20 11:07:04.600155 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:04.599719    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" event={"ID":"7af4e3fa-7494-436d-bdcd-f622a6fdd15b","Type":"ContainerStarted","Data":"681ecd044176d6f0348deafaffaf925f43216d5e77354598bcba41a31b1a98bc"}
Jul 20 11:07:04.600155 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:04.599786    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:07:04.622095 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:04.622049    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" podStartSLOduration=2.872712051 podStartE2EDuration="2m1.62203574s" totalImagesPullingTime="1m54.892463514s" totalInitContainerRuntime="3.856860175s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:07:04.620252989 +0000 UTC m=+4204.458351576" watchObservedRunningTime="2026-07-20 11:07:04.62203574 +0000 UTC m=+4204.460134326"
Jul 20 11:07:05.603520 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:05.603484    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:07:05.604542 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:05.604513    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Jul 20 11:07:06.606845 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:06.606808    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Jul 20 11:07:11.610935 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:11.610903    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:07:11.611756 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:11.611736    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:07:15.075779 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:15.075748    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"]
Jul 20 11:07:15.076243 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:15.076006    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="kserve-container" containerID="cri-o://681ecd044176d6f0348deafaffaf925f43216d5e77354598bcba41a31b1a98bc" gracePeriod=30
Jul 20 11:07:15.076243 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:15.076096    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="kube-rbac-proxy" containerID="cri-o://65613bec1431463af64fdf84f9a8a3e1f540053b1c52c93ef0818e7321f954d3" gracePeriod=30
Jul 20 11:07:15.641913 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:15.641883    2561 generic.go:356] "Generic (PLEG): container finished" podID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerID="65613bec1431463af64fdf84f9a8a3e1f540053b1c52c93ef0818e7321f954d3" exitCode=2
Jul 20 11:07:15.642089 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:15.641932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" event={"ID":"7af4e3fa-7494-436d-bdcd-f622a6fdd15b","Type":"ContainerDied","Data":"65613bec1431463af64fdf84f9a8a3e1f540053b1c52c93ef0818e7321f954d3"}
Jul 20 11:07:16.232139 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:16.232094    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 11:07:16.238254 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:16.238231    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 11:07:16.607352 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:16.607263    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Jul 20 11:07:17.652425 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.652396    2561 generic.go:356] "Generic (PLEG): container finished" podID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerID="681ecd044176d6f0348deafaffaf925f43216d5e77354598bcba41a31b1a98bc" exitCode=0
Jul 20 11:07:17.652820 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.652450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" event={"ID":"7af4e3fa-7494-436d-bdcd-f622a6fdd15b","Type":"ContainerDied","Data":"681ecd044176d6f0348deafaffaf925f43216d5e77354598bcba41a31b1a98bc"}
Jul 20 11:07:17.743321 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.743295    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:07:17.806640 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.806616    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-proxy-tls\") pod \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") "
Jul 20 11:07:17.806786 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.806653    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kserve-provision-location\") pod \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") "
Jul 20 11:07:17.806786 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.806699    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-cabundle-cert\") pod \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") "
Jul 20 11:07:17.806786 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.806729    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kube-rbac-proxy-sar-config\") pod \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") "
Jul 20 11:07:17.806786 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.806778    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drnzz\" (UniqueName: \"kubernetes.io/projected/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kube-api-access-drnzz\") pod \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\" (UID: \"7af4e3fa-7494-436d-bdcd-f622a6fdd15b\") "
Jul 20 11:07:17.807163 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.807084    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7af4e3fa-7494-436d-bdcd-f622a6fdd15b" (UID: "7af4e3fa-7494-436d-bdcd-f622a6fdd15b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:07:17.807163 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.807093    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7af4e3fa-7494-436d-bdcd-f622a6fdd15b" (UID: "7af4e3fa-7494-436d-bdcd-f622a6fdd15b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:07:17.807163 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.807088    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7af4e3fa-7494-436d-bdcd-f622a6fdd15b" (UID: "7af4e3fa-7494-436d-bdcd-f622a6fdd15b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:07:17.808801 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.808778    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kube-api-access-drnzz" (OuterVolumeSpecName: "kube-api-access-drnzz") pod "7af4e3fa-7494-436d-bdcd-f622a6fdd15b" (UID: "7af4e3fa-7494-436d-bdcd-f622a6fdd15b"). InnerVolumeSpecName "kube-api-access-drnzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:07:17.808801 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.808790    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7af4e3fa-7494-436d-bdcd-f622a6fdd15b" (UID: "7af4e3fa-7494-436d-bdcd-f622a6fdd15b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:07:17.907924 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.907901    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:07:17.907924 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.907922    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:07:17.908053 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.907932    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drnzz\" (UniqueName: \"kubernetes.io/projected/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kube-api-access-drnzz\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:07:17.908053 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.907941    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:07:17.908053 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:17.907950    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7af4e3fa-7494-436d-bdcd-f622a6fdd15b-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:07:18.658354 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:18.658318    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg" event={"ID":"7af4e3fa-7494-436d-bdcd-f622a6fdd15b","Type":"ContainerDied","Data":"f792e3d162decc3c7a1e331432c8f0ab6e67256b80dab01ca9b4ec8ca3a44112"}
Jul 20 11:07:18.658354 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:18.658340    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"
Jul 20 11:07:18.658890 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:18.658384    2561 scope.go:117] "RemoveContainer" containerID="65613bec1431463af64fdf84f9a8a3e1f540053b1c52c93ef0818e7321f954d3"
Jul 20 11:07:18.666665 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:18.666649    2561 scope.go:117] "RemoveContainer" containerID="681ecd044176d6f0348deafaffaf925f43216d5e77354598bcba41a31b1a98bc"
Jul 20 11:07:18.673584 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:18.673566    2561 scope.go:117] "RemoveContainer" containerID="bfc07caa4a880470ca37267ddb8173df911ed98a730225b4369e31a1341b2315"
Jul 20 11:07:18.679840 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:18.679815    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"]
Jul 20 11:07:18.687900 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:18.684931    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jprgg"]
Jul 20 11:07:18.765538 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:18.765510    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" path="/var/lib/kubelet/pods/7af4e3fa-7494-436d-bdcd-f622a6fdd15b/volumes"
Jul 20 11:07:20.197619 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197584    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"]
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197840    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kserve-container"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197852    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kserve-container"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197863    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="storage-initializer"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197869    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="storage-initializer"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197875    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="kserve-container"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197880    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="kserve-container"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197892    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kube-rbac-proxy"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197897    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kube-rbac-proxy"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197903    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="kube-rbac-proxy"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197909    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="kube-rbac-proxy"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197915    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="storage-initializer"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197920    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="storage-initializer"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197960    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kube-rbac-proxy"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197966    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="kserve-container"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197972    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="41c1bbd4-d6cd-4ac8-8321-ab25a8a98cc7" containerName="kserve-container"
Jul 20 11:07:20.198025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.197978    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af4e3fa-7494-436d-bdcd-f622a6fdd15b" containerName="kube-rbac-proxy"
Jul 20 11:07:20.230073 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.230038    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"]
Jul 20 11:07:20.230272 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.230197    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.232855 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.232830    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 20 11:07:20.233054 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.232861    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 11:07:20.233279 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.233259    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 20 11:07:20.233401 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.233284    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 11:07:20.233401 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.233340    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 11:07:20.234062 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.234039    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 11:07:20.234200 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.234180    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 11:07:20.324463 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.324426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17578c91-6ca3-486e-a2be-b8cbc22e0617-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.324683 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.324471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpp5n\" (UniqueName: \"kubernetes.io/projected/17578c91-6ca3-486e-a2be-b8cbc22e0617-kube-api-access-rpp5n\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.324683 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.324559    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17578c91-6ca3-486e-a2be-b8cbc22e0617-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.324683 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.324623    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17578c91-6ca3-486e-a2be-b8cbc22e0617-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.324683 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.324659    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17578c91-6ca3-486e-a2be-b8cbc22e0617-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.425846 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.425813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17578c91-6ca3-486e-a2be-b8cbc22e0617-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.425846 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.425847    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpp5n\" (UniqueName: \"kubernetes.io/projected/17578c91-6ca3-486e-a2be-b8cbc22e0617-kube-api-access-rpp5n\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.426107 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.425873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17578c91-6ca3-486e-a2be-b8cbc22e0617-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.426107 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.425901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17578c91-6ca3-486e-a2be-b8cbc22e0617-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.426107 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.425922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17578c91-6ca3-486e-a2be-b8cbc22e0617-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.426107 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:07:20.425988    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
Jul 20 11:07:20.426107 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:07:20.426098    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17578c91-6ca3-486e-a2be-b8cbc22e0617-proxy-tls podName:17578c91-6ca3-486e-a2be-b8cbc22e0617 nodeName:}" failed. No retries permitted until 2026-07-20 11:07:20.926075932 +0000 UTC m=+4220.764174502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/17578c91-6ca3-486e-a2be-b8cbc22e0617-proxy-tls") pod "isvc-xgboost-predictor-7566d65bf7-pftpp" (UID: "17578c91-6ca3-486e-a2be-b8cbc22e0617") : secret "isvc-xgboost-predictor-serving-cert" not found
Jul 20 11:07:20.426407 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.426321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17578c91-6ca3-486e-a2be-b8cbc22e0617-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.426556 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.426536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17578c91-6ca3-486e-a2be-b8cbc22e0617-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.426628 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.426612    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17578c91-6ca3-486e-a2be-b8cbc22e0617-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.434465 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.434441    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpp5n\" (UniqueName: \"kubernetes.io/projected/17578c91-6ca3-486e-a2be-b8cbc22e0617-kube-api-access-rpp5n\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.929443 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.929409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17578c91-6ca3-486e-a2be-b8cbc22e0617-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:20.931748 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:20.931720    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17578c91-6ca3-486e-a2be-b8cbc22e0617-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-pftpp\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:21.140657 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:21.140618    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:21.326879 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:21.326854    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"]
Jul 20 11:07:21.329404 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:07:21.329378    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17578c91_6ca3_486e_a2be_b8cbc22e0617.slice/crio-0c602abea207c11988e516f36110d3729ddc0ae26d116cfb1fefd58a3788bb62 WatchSource:0}: Error finding container 0c602abea207c11988e516f36110d3729ddc0ae26d116cfb1fefd58a3788bb62: Status 404 returned error can't find the container with id 0c602abea207c11988e516f36110d3729ddc0ae26d116cfb1fefd58a3788bb62
Jul 20 11:07:21.671808 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:21.671779    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" event={"ID":"17578c91-6ca3-486e-a2be-b8cbc22e0617","Type":"ContainerStarted","Data":"9fe318cef5994d88a95abd23794a1f5f93f902bf2ff84946ac14c6c3ec67edc9"}
Jul 20 11:07:21.671949 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:21.671841    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" event={"ID":"17578c91-6ca3-486e-a2be-b8cbc22e0617","Type":"ContainerStarted","Data":"0c602abea207c11988e516f36110d3729ddc0ae26d116cfb1fefd58a3788bb62"}
Jul 20 11:07:25.687592 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:25.687557    2561 generic.go:356] "Generic (PLEG): container finished" podID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerID="9fe318cef5994d88a95abd23794a1f5f93f902bf2ff84946ac14c6c3ec67edc9" exitCode=0
Jul 20 11:07:25.688011 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:25.687607    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" event={"ID":"17578c91-6ca3-486e-a2be-b8cbc22e0617","Type":"ContainerDied","Data":"9fe318cef5994d88a95abd23794a1f5f93f902bf2ff84946ac14c6c3ec67edc9"}
Jul 20 11:07:44.768492 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:44.768464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" event={"ID":"17578c91-6ca3-486e-a2be-b8cbc22e0617","Type":"ContainerStarted","Data":"9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9"}
Jul 20 11:07:45.774190 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:45.774145    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" event={"ID":"17578c91-6ca3-486e-a2be-b8cbc22e0617","Type":"ContainerStarted","Data":"9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017"}
Jul 20 11:07:45.774591 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:45.774230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:45.793821 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:45.793767    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" podStartSLOduration=3.364460583 podStartE2EDuration="25.793748687s" totalImagesPullingTime="18.868901898s" totalInitContainerRuntime="3.560386206s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:07:45.793050766 +0000 UTC m=+4245.631149354" watchObservedRunningTime="2026-07-20 11:07:45.793748687 +0000 UTC m=+4245.631847273"
Jul 20 11:07:46.777278 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:46.777242    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:46.778363 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:46.778338    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 20 11:07:47.781359 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:47.781313    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 20 11:07:52.786375 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:52.786346    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:07:52.786829 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:07:52.786793    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 20 11:08:02.787715 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:08:02.787679    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 20 11:08:12.787484 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:08:12.787444    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 20 11:08:22.786815 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:08:22.786774    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 20 11:08:28.925312 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:08:28.925280    2561 generic.go:356] "Generic (PLEG): container finished" podID="5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263" containerID="7d028ed9d09467d48025dc8daf451b8063ca6652c9ca46c878d7a057187c5b84" exitCode=1
Jul 20 11:08:28.925715 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:08:28.925357    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-w4qrj" event={"ID":"5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263","Type":"ContainerDied","Data":"7d028ed9d09467d48025dc8daf451b8063ca6652c9ca46c878d7a057187c5b84"}
Jul 20 11:08:28.925761 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:08:28.925752    2561 scope.go:117] "RemoveContainer" containerID="7d028ed9d09467d48025dc8daf451b8063ca6652c9ca46c878d7a057187c5b84"
Jul 20 11:08:29.930593 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:08:29.930501    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-w4qrj" event={"ID":"5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263","Type":"ContainerStarted","Data":"0ad8ec62ee4d0f2918dbc9c489929fc41eeabe85b552252d3092ce319fb31ace"}
Jul 20 11:08:29.930980 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:08:29.930708    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-645f4f9567-w4qrj"
Jul 20 11:08:32.787580 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:08:32.787537    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 20 11:08:42.787921 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:08:42.787893    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:09:00.939036 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:00.938968    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-645f4f9567-w4qrj"
Jul 20 11:09:06.199634 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.199598    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"]
Jul 20 11:09:06.202129 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.202099    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.204709 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.204670    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 20 11:09:06.204830 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.204690    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 20 11:09:06.213925 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.213900    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"]
Jul 20 11:09:06.342018 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.341985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2865d1b-2e56-40db-9021-a81662a996ca-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.342203 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.342025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2865d1b-2e56-40db-9021-a81662a996ca-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.342203 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.342050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2865d1b-2e56-40db-9021-a81662a996ca-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.342203 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.342136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2892\" (UniqueName: \"kubernetes.io/projected/a2865d1b-2e56-40db-9021-a81662a996ca-kube-api-access-t2892\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.342203 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.342182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2865d1b-2e56-40db-9021-a81662a996ca-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.443387 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.443355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2865d1b-2e56-40db-9021-a81662a996ca-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.443532 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.443408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2865d1b-2e56-40db-9021-a81662a996ca-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.443532 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.443429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2865d1b-2e56-40db-9021-a81662a996ca-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.443532 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.443449    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2865d1b-2e56-40db-9021-a81662a996ca-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.443532 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.443470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2892\" (UniqueName: \"kubernetes.io/projected/a2865d1b-2e56-40db-9021-a81662a996ca-kube-api-access-t2892\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.443861 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.443839    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2865d1b-2e56-40db-9021-a81662a996ca-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.444158 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.444101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2865d1b-2e56-40db-9021-a81662a996ca-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.444268 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.444205    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2865d1b-2e56-40db-9021-a81662a996ca-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.445899 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.445880    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2865d1b-2e56-40db-9021-a81662a996ca-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.452468 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.452417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2892\" (UniqueName: \"kubernetes.io/projected/a2865d1b-2e56-40db-9021-a81662a996ca-kube-api-access-t2892\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-jpp7j\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.511984 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.511949    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:06.638597 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:06.638571    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"]
Jul 20 11:09:06.641357 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:09:06.641323    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2865d1b_2e56_40db_9021_a81662a996ca.slice/crio-132c965fe7c59440e97a3f298b686f325c17f2fc0948abdc476f00fe17694ff8 WatchSource:0}: Error finding container 132c965fe7c59440e97a3f298b686f325c17f2fc0948abdc476f00fe17694ff8: Status 404 returned error can't find the container with id 132c965fe7c59440e97a3f298b686f325c17f2fc0948abdc476f00fe17694ff8
Jul 20 11:09:07.067446 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:07.067411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" event={"ID":"a2865d1b-2e56-40db-9021-a81662a996ca","Type":"ContainerStarted","Data":"c33f668a400f5d31ca7910f795057a3a4105d3eabc84f5745aafd82ac63546ae"}
Jul 20 11:09:07.067612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:07.067474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" event={"ID":"a2865d1b-2e56-40db-9021-a81662a996ca","Type":"ContainerStarted","Data":"132c965fe7c59440e97a3f298b686f325c17f2fc0948abdc476f00fe17694ff8"}
Jul 20 11:09:10.859124 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:10.859072    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"]
Jul 20 11:09:10.859563 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:10.859379    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kserve-container" containerID="cri-o://9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9" gracePeriod=30
Jul 20 11:09:10.859563 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:10.859407    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kube-rbac-proxy" containerID="cri-o://9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017" gracePeriod=30
Jul 20 11:09:11.084457 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:11.084426    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2865d1b-2e56-40db-9021-a81662a996ca" containerID="c33f668a400f5d31ca7910f795057a3a4105d3eabc84f5745aafd82ac63546ae" exitCode=0
Jul 20 11:09:11.084608 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:11.084505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" event={"ID":"a2865d1b-2e56-40db-9021-a81662a996ca","Type":"ContainerDied","Data":"c33f668a400f5d31ca7910f795057a3a4105d3eabc84f5745aafd82ac63546ae"}
Jul 20 11:09:11.086887 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:11.086865    2561 generic.go:356] "Generic (PLEG): container finished" podID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerID="9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017" exitCode=2
Jul 20 11:09:11.086974 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:11.086949    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" event={"ID":"17578c91-6ca3-486e-a2be-b8cbc22e0617","Type":"ContainerDied","Data":"9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017"}
Jul 20 11:09:12.092921 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:12.092886    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" event={"ID":"a2865d1b-2e56-40db-9021-a81662a996ca","Type":"ContainerStarted","Data":"b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b"}
Jul 20 11:09:12.093550 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:12.092955    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" event={"ID":"a2865d1b-2e56-40db-9021-a81662a996ca","Type":"ContainerStarted","Data":"e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3"}
Jul 20 11:09:12.093550 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:12.093187    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:12.114294 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:12.114227    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podStartSLOduration=2.619513371 podStartE2EDuration="6.114214479s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.494701108s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:09:12.114035676 +0000 UTC m=+4331.952134271" watchObservedRunningTime="2026-07-20 11:09:12.114214479 +0000 UTC m=+4331.952313064"
Jul 20 11:09:12.782418 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:12.782383    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Jul 20 11:09:12.786741 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:12.786714    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 20 11:09:13.097229 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:13.097144    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:13.098232 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:13.098205    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Jul 20 11:09:14.101071 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.101026    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Jul 20 11:09:14.390594 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.390572    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:09:14.504886 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.504847    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17578c91-6ca3-486e-a2be-b8cbc22e0617-cabundle-cert\") pod \"17578c91-6ca3-486e-a2be-b8cbc22e0617\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") "
Jul 20 11:09:14.505075 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.504897    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17578c91-6ca3-486e-a2be-b8cbc22e0617-proxy-tls\") pod \"17578c91-6ca3-486e-a2be-b8cbc22e0617\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") "
Jul 20 11:09:14.505075 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.504928    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17578c91-6ca3-486e-a2be-b8cbc22e0617-kube-rbac-proxy-sar-config\") pod \"17578c91-6ca3-486e-a2be-b8cbc22e0617\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") "
Jul 20 11:09:14.505075 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.504958    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpp5n\" (UniqueName: \"kubernetes.io/projected/17578c91-6ca3-486e-a2be-b8cbc22e0617-kube-api-access-rpp5n\") pod \"17578c91-6ca3-486e-a2be-b8cbc22e0617\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") "
Jul 20 11:09:14.505075 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.505017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17578c91-6ca3-486e-a2be-b8cbc22e0617-kserve-provision-location\") pod \"17578c91-6ca3-486e-a2be-b8cbc22e0617\" (UID: \"17578c91-6ca3-486e-a2be-b8cbc22e0617\") "
Jul 20 11:09:14.505346 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.505319    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17578c91-6ca3-486e-a2be-b8cbc22e0617-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "17578c91-6ca3-486e-a2be-b8cbc22e0617" (UID: "17578c91-6ca3-486e-a2be-b8cbc22e0617"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:09:14.505386 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.505362    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17578c91-6ca3-486e-a2be-b8cbc22e0617-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17578c91-6ca3-486e-a2be-b8cbc22e0617" (UID: "17578c91-6ca3-486e-a2be-b8cbc22e0617"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:09:14.505423 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.505396    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17578c91-6ca3-486e-a2be-b8cbc22e0617-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17578c91-6ca3-486e-a2be-b8cbc22e0617" (UID: "17578c91-6ca3-486e-a2be-b8cbc22e0617"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:09:14.506999 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.506971    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17578c91-6ca3-486e-a2be-b8cbc22e0617-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17578c91-6ca3-486e-a2be-b8cbc22e0617" (UID: "17578c91-6ca3-486e-a2be-b8cbc22e0617"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:09:14.507161 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.507001    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17578c91-6ca3-486e-a2be-b8cbc22e0617-kube-api-access-rpp5n" (OuterVolumeSpecName: "kube-api-access-rpp5n") pod "17578c91-6ca3-486e-a2be-b8cbc22e0617" (UID: "17578c91-6ca3-486e-a2be-b8cbc22e0617"). InnerVolumeSpecName "kube-api-access-rpp5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:09:14.606306 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.606264    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17578c91-6ca3-486e-a2be-b8cbc22e0617-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:09:14.606306 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.606296    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17578c91-6ca3-486e-a2be-b8cbc22e0617-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:09:14.606306 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.606308    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17578c91-6ca3-486e-a2be-b8cbc22e0617-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:09:14.606306 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.606316    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17578c91-6ca3-486e-a2be-b8cbc22e0617-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:09:14.606571 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:14.606325    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpp5n\" (UniqueName: \"kubernetes.io/projected/17578c91-6ca3-486e-a2be-b8cbc22e0617-kube-api-access-rpp5n\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:09:15.107339 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.107252    2561 generic.go:356] "Generic (PLEG): container finished" podID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerID="9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9" exitCode=0
Jul 20 11:09:15.107339 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.107307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" event={"ID":"17578c91-6ca3-486e-a2be-b8cbc22e0617","Type":"ContainerDied","Data":"9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9"}
Jul 20 11:09:15.107339 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.107320    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"
Jul 20 11:09:15.107850 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.107353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp" event={"ID":"17578c91-6ca3-486e-a2be-b8cbc22e0617","Type":"ContainerDied","Data":"0c602abea207c11988e516f36110d3729ddc0ae26d116cfb1fefd58a3788bb62"}
Jul 20 11:09:15.107850 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.107379    2561 scope.go:117] "RemoveContainer" containerID="9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017"
Jul 20 11:09:15.115226 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.115206    2561 scope.go:117] "RemoveContainer" containerID="9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9"
Jul 20 11:09:15.122275 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.122258    2561 scope.go:117] "RemoveContainer" containerID="9fe318cef5994d88a95abd23794a1f5f93f902bf2ff84946ac14c6c3ec67edc9"
Jul 20 11:09:15.127660 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.127639    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"]
Jul 20 11:09:15.129846 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.129828    2561 scope.go:117] "RemoveContainer" containerID="9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017"
Jul 20 11:09:15.130076 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:09:15.130059    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017\": container with ID starting with 9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017 not found: ID does not exist" containerID="9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017"
Jul 20 11:09:15.130215 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.130086    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017"} err="failed to get container status \"9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017\": rpc error: code = NotFound desc = could not find container \"9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017\": container with ID starting with 9ffb811b550ed66facf1cdeff148cd3a8067c5a21e53140f1387f7d3d2892017 not found: ID does not exist"
Jul 20 11:09:15.130215 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.130102    2561 scope.go:117] "RemoveContainer" containerID="9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9"
Jul 20 11:09:15.130376 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:09:15.130361    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9\": container with ID starting with 9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9 not found: ID does not exist" containerID="9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9"
Jul 20 11:09:15.130414 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.130381    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9"} err="failed to get container status \"9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9\": rpc error: code = NotFound desc = could not find container \"9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9\": container with ID starting with 9a4feb2e806591badedb318609cacd00d2f5d9655414082e8693a491a439f1e9 not found: ID does not exist"
Jul 20 11:09:15.130414 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.130394    2561 scope.go:117] "RemoveContainer" containerID="9fe318cef5994d88a95abd23794a1f5f93f902bf2ff84946ac14c6c3ec67edc9"
Jul 20 11:09:15.130578 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:09:15.130562    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fe318cef5994d88a95abd23794a1f5f93f902bf2ff84946ac14c6c3ec67edc9\": container with ID starting with 9fe318cef5994d88a95abd23794a1f5f93f902bf2ff84946ac14c6c3ec67edc9 not found: ID does not exist" containerID="9fe318cef5994d88a95abd23794a1f5f93f902bf2ff84946ac14c6c3ec67edc9"
Jul 20 11:09:15.130614 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.130583    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fe318cef5994d88a95abd23794a1f5f93f902bf2ff84946ac14c6c3ec67edc9"} err="failed to get container status \"9fe318cef5994d88a95abd23794a1f5f93f902bf2ff84946ac14c6c3ec67edc9\": rpc error: code = NotFound desc = could not find container \"9fe318cef5994d88a95abd23794a1f5f93f902bf2ff84946ac14c6c3ec67edc9\": container with ID starting with 9fe318cef5994d88a95abd23794a1f5f93f902bf2ff84946ac14c6c3ec67edc9 not found: ID does not exist"
Jul 20 11:09:15.133446 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:15.133426    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-pftpp"]
Jul 20 11:09:16.765917 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:16.765884    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" path="/var/lib/kubelet/pods/17578c91-6ca3-486e-a2be-b8cbc22e0617/volumes"
Jul 20 11:09:19.106718 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:19.106688    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:09:19.107361 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:19.107332    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Jul 20 11:09:29.107821 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:29.107785    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Jul 20 11:09:39.107442 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:39.107402    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Jul 20 11:09:49.107286 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:49.107243    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Jul 20 11:09:59.108229 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:09:59.108186    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Jul 20 11:10:09.107625 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:09.107585    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Jul 20 11:10:19.108277 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:19.108248    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:10:26.299552 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:26.299477    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"]
Jul 20 11:10:26.299987 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:26.299713    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container" containerID="cri-o://e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3" gracePeriod=30
Jul 20 11:10:26.299987 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:26.299778    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kube-rbac-proxy" containerID="cri-o://b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b" gracePeriod=30
Jul 20 11:10:27.375876 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:27.375845    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2865d1b-2e56-40db-9021-a81662a996ca" containerID="b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b" exitCode=2
Jul 20 11:10:27.376273 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:27.375885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" event={"ID":"a2865d1b-2e56-40db-9021-a81662a996ca","Type":"ContainerDied","Data":"b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b"}
Jul 20 11:10:29.101569 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.101524    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Jul 20 11:10:29.107604 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.107578    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Jul 20 11:10:29.745045 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.745017    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:10:29.859693 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.859603    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2865d1b-2e56-40db-9021-a81662a996ca-proxy-tls\") pod \"a2865d1b-2e56-40db-9021-a81662a996ca\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") "
Jul 20 11:10:29.859693 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.859670    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2892\" (UniqueName: \"kubernetes.io/projected/a2865d1b-2e56-40db-9021-a81662a996ca-kube-api-access-t2892\") pod \"a2865d1b-2e56-40db-9021-a81662a996ca\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") "
Jul 20 11:10:29.859894 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.859704    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2865d1b-2e56-40db-9021-a81662a996ca-cabundle-cert\") pod \"a2865d1b-2e56-40db-9021-a81662a996ca\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") "
Jul 20 11:10:29.859894 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.859725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2865d1b-2e56-40db-9021-a81662a996ca-kube-rbac-proxy-sar-config\") pod \"a2865d1b-2e56-40db-9021-a81662a996ca\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") "
Jul 20 11:10:29.859894 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.859795    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2865d1b-2e56-40db-9021-a81662a996ca-kserve-provision-location\") pod \"a2865d1b-2e56-40db-9021-a81662a996ca\" (UID: \"a2865d1b-2e56-40db-9021-a81662a996ca\") "
Jul 20 11:10:29.860087 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.860061    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2865d1b-2e56-40db-9021-a81662a996ca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2865d1b-2e56-40db-9021-a81662a996ca" (UID: "a2865d1b-2e56-40db-9021-a81662a996ca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:10:29.860224 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.860204    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2865d1b-2e56-40db-9021-a81662a996ca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2865d1b-2e56-40db-9021-a81662a996ca" (UID: "a2865d1b-2e56-40db-9021-a81662a996ca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:10:29.860298 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.860223    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2865d1b-2e56-40db-9021-a81662a996ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2865d1b-2e56-40db-9021-a81662a996ca" (UID: "a2865d1b-2e56-40db-9021-a81662a996ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:10:29.861773 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.861752    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2865d1b-2e56-40db-9021-a81662a996ca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2865d1b-2e56-40db-9021-a81662a996ca" (UID: "a2865d1b-2e56-40db-9021-a81662a996ca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:10:29.861881 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.861872    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2865d1b-2e56-40db-9021-a81662a996ca-kube-api-access-t2892" (OuterVolumeSpecName: "kube-api-access-t2892") pod "a2865d1b-2e56-40db-9021-a81662a996ca" (UID: "a2865d1b-2e56-40db-9021-a81662a996ca"). InnerVolumeSpecName "kube-api-access-t2892". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:10:29.960388 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.960356    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2865d1b-2e56-40db-9021-a81662a996ca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:10:29.960388 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.960385    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2865d1b-2e56-40db-9021-a81662a996ca-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:10:29.960388 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.960394    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2865d1b-2e56-40db-9021-a81662a996ca-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:10:29.960631 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.960403    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2892\" (UniqueName: \"kubernetes.io/projected/a2865d1b-2e56-40db-9021-a81662a996ca-kube-api-access-t2892\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:10:29.960631 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:29.960412    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2865d1b-2e56-40db-9021-a81662a996ca-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:10:30.387798 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.387758    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2865d1b-2e56-40db-9021-a81662a996ca" containerID="e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3" exitCode=0
Jul 20 11:10:30.388272 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.387841    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"
Jul 20 11:10:30.388272 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.387850    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" event={"ID":"a2865d1b-2e56-40db-9021-a81662a996ca","Type":"ContainerDied","Data":"e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3"}
Jul 20 11:10:30.388272 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.387970    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j" event={"ID":"a2865d1b-2e56-40db-9021-a81662a996ca","Type":"ContainerDied","Data":"132c965fe7c59440e97a3f298b686f325c17f2fc0948abdc476f00fe17694ff8"}
Jul 20 11:10:30.388272 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.388011    2561 scope.go:117] "RemoveContainer" containerID="b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b"
Jul 20 11:10:30.396334 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.396313    2561 scope.go:117] "RemoveContainer" containerID="e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3"
Jul 20 11:10:30.403256 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.403238    2561 scope.go:117] "RemoveContainer" containerID="c33f668a400f5d31ca7910f795057a3a4105d3eabc84f5745aafd82ac63546ae"
Jul 20 11:10:30.409050 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.409026    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"]
Jul 20 11:10:30.410947 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.410928    2561 scope.go:117] "RemoveContainer" containerID="b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b"
Jul 20 11:10:30.411395 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:10:30.411370    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b\": container with ID starting with b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b not found: ID does not exist" containerID="b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b"
Jul 20 11:10:30.411499 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.411406    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b"} err="failed to get container status \"b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b\": rpc error: code = NotFound desc = could not find container \"b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b\": container with ID starting with b58cc3fd134542081eb87eda9d3a4dc4786c287a7a6e58ada101d1ec5aff9a7b not found: ID does not exist"
Jul 20 11:10:30.411499 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.411427    2561 scope.go:117] "RemoveContainer" containerID="e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3"
Jul 20 11:10:30.411682 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:10:30.411664    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3\": container with ID starting with e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3 not found: ID does not exist" containerID="e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3"
Jul 20 11:10:30.411751 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.411692    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3"} err="failed to get container status \"e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3\": rpc error: code = NotFound desc = could not find container \"e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3\": container with ID starting with e7ad8b48df257d01854ac7fcbb499db14230d8998fb601544d3d3454284d4ab3 not found: ID does not exist"
Jul 20 11:10:30.411751 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.411714    2561 scope.go:117] "RemoveContainer" containerID="c33f668a400f5d31ca7910f795057a3a4105d3eabc84f5745aafd82ac63546ae"
Jul 20 11:10:30.411949 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:10:30.411934    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33f668a400f5d31ca7910f795057a3a4105d3eabc84f5745aafd82ac63546ae\": container with ID starting with c33f668a400f5d31ca7910f795057a3a4105d3eabc84f5745aafd82ac63546ae not found: ID does not exist" containerID="c33f668a400f5d31ca7910f795057a3a4105d3eabc84f5745aafd82ac63546ae"
Jul 20 11:10:30.411999 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.411955    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33f668a400f5d31ca7910f795057a3a4105d3eabc84f5745aafd82ac63546ae"} err="failed to get container status \"c33f668a400f5d31ca7910f795057a3a4105d3eabc84f5745aafd82ac63546ae\": rpc error: code = NotFound desc = could not find container \"c33f668a400f5d31ca7910f795057a3a4105d3eabc84f5745aafd82ac63546ae\": container with ID starting with c33f668a400f5d31ca7910f795057a3a4105d3eabc84f5745aafd82ac63546ae not found: ID does not exist"
Jul 20 11:10:30.412352 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.412332    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-jpp7j"]
Jul 20 11:10:30.764871 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:30.764785    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" path="/var/lib/kubelet/pods/a2865d1b-2e56-40db-9021-a81662a996ca/volumes"
Jul 20 11:10:31.464861 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.464827    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"]
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465069    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="storage-initializer"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465080    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="storage-initializer"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465092    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kube-rbac-proxy"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465097    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kube-rbac-proxy"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465103    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465123    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465131    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kube-rbac-proxy"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465136    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kube-rbac-proxy"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465143    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kserve-container"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465148    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kserve-container"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465164    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="storage-initializer"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465169    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="storage-initializer"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465212    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kserve-container"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465219    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17578c91-6ca3-486e-a2be-b8cbc22e0617" containerName="kube-rbac-proxy"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465225    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kube-rbac-proxy"
Jul 20 11:10:31.465285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.465231    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2865d1b-2e56-40db-9021-a81662a996ca" containerName="kserve-container"
Jul 20 11:10:31.469778 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.469755    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.472154 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.472135    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 20 11:10:31.472241 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.472185    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 11:10:31.472737 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.472719    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 11:10:31.472812 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.472759    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 20 11:10:31.472812 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.472777    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 11:10:31.472812 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.472791    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 11:10:31.473804 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.473786    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 11:10:31.478561 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.478541    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"]
Jul 20 11:10:31.574874 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.574848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhdbf\" (UniqueName: \"kubernetes.io/projected/d1b90694-e22f-47fe-b2b0-d46de12ad344-kube-api-access-xhdbf\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.575031 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.574887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1b90694-e22f-47fe-b2b0-d46de12ad344-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.575031 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.574930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1b90694-e22f-47fe-b2b0-d46de12ad344-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.575031 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.575011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1b90694-e22f-47fe-b2b0-d46de12ad344-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.575202 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.575051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1b90694-e22f-47fe-b2b0-d46de12ad344-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.675412 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.675366    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1b90694-e22f-47fe-b2b0-d46de12ad344-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.675412 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.675415    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1b90694-e22f-47fe-b2b0-d46de12ad344-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.675664 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.675442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhdbf\" (UniqueName: \"kubernetes.io/projected/d1b90694-e22f-47fe-b2b0-d46de12ad344-kube-api-access-xhdbf\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.675664 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.675462    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1b90694-e22f-47fe-b2b0-d46de12ad344-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.675664 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.675493    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1b90694-e22f-47fe-b2b0-d46de12ad344-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.675822 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.675793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1b90694-e22f-47fe-b2b0-d46de12ad344-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.676264 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.676242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1b90694-e22f-47fe-b2b0-d46de12ad344-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.676325 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.676242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1b90694-e22f-47fe-b2b0-d46de12ad344-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.677861 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.677843    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1b90694-e22f-47fe-b2b0-d46de12ad344-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.683753 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.683731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhdbf\" (UniqueName: \"kubernetes.io/projected/d1b90694-e22f-47fe-b2b0-d46de12ad344-kube-api-access-xhdbf\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.780894 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.780802    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:31.904325 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:31.904300    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"]
Jul 20 11:10:31.906775 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:10:31.906745    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b90694_e22f_47fe_b2b0_d46de12ad344.slice/crio-0a2a58a0321fe3f62eab42831cc14232e126190da025244fec74d03e56c9647e WatchSource:0}: Error finding container 0a2a58a0321fe3f62eab42831cc14232e126190da025244fec74d03e56c9647e: Status 404 returned error can't find the container with id 0a2a58a0321fe3f62eab42831cc14232e126190da025244fec74d03e56c9647e
Jul 20 11:10:32.398801 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:32.398766    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" event={"ID":"d1b90694-e22f-47fe-b2b0-d46de12ad344","Type":"ContainerStarted","Data":"4678ea9bfd3c1daefc985dd1996d6ddd348f0d0d51ef239f523f0fee0ea12a7e"}
Jul 20 11:10:32.398972 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:32.398821    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" event={"ID":"d1b90694-e22f-47fe-b2b0-d46de12ad344","Type":"ContainerStarted","Data":"0a2a58a0321fe3f62eab42831cc14232e126190da025244fec74d03e56c9647e"}
Jul 20 11:10:36.418319 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:36.418281    2561 generic.go:356] "Generic (PLEG): container finished" podID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerID="4678ea9bfd3c1daefc985dd1996d6ddd348f0d0d51ef239f523f0fee0ea12a7e" exitCode=0
Jul 20 11:10:36.418703 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:36.418316    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" event={"ID":"d1b90694-e22f-47fe-b2b0-d46de12ad344","Type":"ContainerDied","Data":"4678ea9bfd3c1daefc985dd1996d6ddd348f0d0d51ef239f523f0fee0ea12a7e"}
Jul 20 11:10:37.428924 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:37.428879    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" event={"ID":"d1b90694-e22f-47fe-b2b0-d46de12ad344","Type":"ContainerStarted","Data":"a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc"}
Jul 20 11:10:37.429326 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:37.429005    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" event={"ID":"d1b90694-e22f-47fe-b2b0-d46de12ad344","Type":"ContainerStarted","Data":"a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7"}
Jul 20 11:10:37.429430 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:37.429399    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:37.429531 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:37.429514    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:10:37.449509 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:37.449458    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" podStartSLOduration=2.858147536 podStartE2EDuration="6.449443999s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.591296463s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:10:37.447573097 +0000 UTC m=+4417.285671682" watchObservedRunningTime="2026-07-20 11:10:37.449443999 +0000 UTC m=+4417.287542583"
Jul 20 11:10:43.438188 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:10:43.438160    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:11:13.455695 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:13.455654    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 20 11:11:23.441547 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:23.441517    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:11:31.556695 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:31.556658    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"]
Jul 20 11:11:31.557189 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:31.556955    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="kserve-container" containerID="cri-o://a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7" gracePeriod=30
Jul 20 11:11:31.557189 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:31.556968    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="kube-rbac-proxy" containerID="cri-o://a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc" gracePeriod=30
Jul 20 11:11:32.634077 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:32.634042    2561 generic.go:356] "Generic (PLEG): container finished" podID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerID="a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc" exitCode=2
Jul 20 11:11:32.634471 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:32.634130    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" event={"ID":"d1b90694-e22f-47fe-b2b0-d46de12ad344","Type":"ContainerDied","Data":"a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc"}
Jul 20 11:11:33.434374 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:33.434331    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.65:8643/healthz\": dial tcp 10.132.0.65:8643: connect: connection refused"
Jul 20 11:11:34.479288 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:34.479235    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.65:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 11:11:36.725992 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.725958    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"]
Jul 20 11:11:36.729807 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.729783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.732367 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.732339    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 20 11:11:36.732510 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.732388    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 20 11:11:36.740438 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.740407    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"]
Jul 20 11:11:36.851008 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.850973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edeba85e-0ab9-47da-aa7b-a8f228c969bd-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.851200 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.851020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.851200 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.851170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.851307 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.851206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edeba85e-0ab9-47da-aa7b-a8f228c969bd-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.851307 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.851228    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhdc4\" (UniqueName: \"kubernetes.io/projected/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kube-api-access-rhdc4\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.951590 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.951556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.951764 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.951604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.951764 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.951629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edeba85e-0ab9-47da-aa7b-a8f228c969bd-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.951764 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.951646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhdc4\" (UniqueName: \"kubernetes.io/projected/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kube-api-access-rhdc4\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.951764 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.951682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edeba85e-0ab9-47da-aa7b-a8f228c969bd-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.952167 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.952136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.952305 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.952253    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.952369 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.952351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edeba85e-0ab9-47da-aa7b-a8f228c969bd-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.953979 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.953952    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edeba85e-0ab9-47da-aa7b-a8f228c969bd-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:36.960474 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:36.960453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhdc4\" (UniqueName: \"kubernetes.io/projected/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kube-api-access-rhdc4\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w7rwz\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:37.042517 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:37.042452    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:37.171312 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:37.171280    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"]
Jul 20 11:11:37.174284 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:11:37.174255    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedeba85e_0ab9_47da_aa7b_a8f228c969bd.slice/crio-2340fe59c458aba0fcb69bd66316f2ce033feb6d1275d878bd82031cc6d1e9de WatchSource:0}: Error finding container 2340fe59c458aba0fcb69bd66316f2ce033feb6d1275d878bd82031cc6d1e9de: Status 404 returned error can't find the container with id 2340fe59c458aba0fcb69bd66316f2ce033feb6d1275d878bd82031cc6d1e9de
Jul 20 11:11:37.653932 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:37.653900    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" event={"ID":"edeba85e-0ab9-47da-aa7b-a8f228c969bd","Type":"ContainerStarted","Data":"928c253bca6c278fd08d3f1db70fc5fabd1b63aff4af8a08de991e384161cf51"}
Jul 20 11:11:37.654104 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:37.653954    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" event={"ID":"edeba85e-0ab9-47da-aa7b-a8f228c969bd","Type":"ContainerStarted","Data":"2340fe59c458aba0fcb69bd66316f2ce033feb6d1275d878bd82031cc6d1e9de"}
Jul 20 11:11:38.505874 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.505852    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:11:38.566764 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.566734    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhdbf\" (UniqueName: \"kubernetes.io/projected/d1b90694-e22f-47fe-b2b0-d46de12ad344-kube-api-access-xhdbf\") pod \"d1b90694-e22f-47fe-b2b0-d46de12ad344\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") "
Jul 20 11:11:38.566932 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.566783    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1b90694-e22f-47fe-b2b0-d46de12ad344-cabundle-cert\") pod \"d1b90694-e22f-47fe-b2b0-d46de12ad344\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") "
Jul 20 11:11:38.566932 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.566804    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1b90694-e22f-47fe-b2b0-d46de12ad344-kserve-provision-location\") pod \"d1b90694-e22f-47fe-b2b0-d46de12ad344\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") "
Jul 20 11:11:38.566932 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.566844    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1b90694-e22f-47fe-b2b0-d46de12ad344-kube-rbac-proxy-sar-config\") pod \"d1b90694-e22f-47fe-b2b0-d46de12ad344\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") "
Jul 20 11:11:38.566932 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.566862    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1b90694-e22f-47fe-b2b0-d46de12ad344-proxy-tls\") pod \"d1b90694-e22f-47fe-b2b0-d46de12ad344\" (UID: \"d1b90694-e22f-47fe-b2b0-d46de12ad344\") "
Jul 20 11:11:38.567197 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.567081    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b90694-e22f-47fe-b2b0-d46de12ad344-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1b90694-e22f-47fe-b2b0-d46de12ad344" (UID: "d1b90694-e22f-47fe-b2b0-d46de12ad344"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:38.567197 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.567160    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b90694-e22f-47fe-b2b0-d46de12ad344-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d1b90694-e22f-47fe-b2b0-d46de12ad344" (UID: "d1b90694-e22f-47fe-b2b0-d46de12ad344"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:11:38.567279 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.567194    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b90694-e22f-47fe-b2b0-d46de12ad344-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1b90694-e22f-47fe-b2b0-d46de12ad344" (UID: "d1b90694-e22f-47fe-b2b0-d46de12ad344"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:11:38.568812 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.568790    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b90694-e22f-47fe-b2b0-d46de12ad344-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1b90694-e22f-47fe-b2b0-d46de12ad344" (UID: "d1b90694-e22f-47fe-b2b0-d46de12ad344"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:11:38.568926 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.568841    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b90694-e22f-47fe-b2b0-d46de12ad344-kube-api-access-xhdbf" (OuterVolumeSpecName: "kube-api-access-xhdbf") pod "d1b90694-e22f-47fe-b2b0-d46de12ad344" (UID: "d1b90694-e22f-47fe-b2b0-d46de12ad344"). InnerVolumeSpecName "kube-api-access-xhdbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:11:38.659600 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.659569    2561 generic.go:356] "Generic (PLEG): container finished" podID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerID="a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7" exitCode=0
Jul 20 11:11:38.659733 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.659634    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"
Jul 20 11:11:38.659733 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.659650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" event={"ID":"d1b90694-e22f-47fe-b2b0-d46de12ad344","Type":"ContainerDied","Data":"a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7"}
Jul 20 11:11:38.659733 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.659708    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" event={"ID":"d1b90694-e22f-47fe-b2b0-d46de12ad344","Type":"ContainerDied","Data":"0a2a58a0321fe3f62eab42831cc14232e126190da025244fec74d03e56c9647e"}
Jul 20 11:11:38.659846 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.659737    2561 scope.go:117] "RemoveContainer" containerID="a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc"
Jul 20 11:11:38.667464 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.667437    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1b90694-e22f-47fe-b2b0-d46de12ad344-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:11:38.667585 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.667469    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1b90694-e22f-47fe-b2b0-d46de12ad344-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:11:38.667585 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.667484    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1b90694-e22f-47fe-b2b0-d46de12ad344-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:11:38.667585 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.667497    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1b90694-e22f-47fe-b2b0-d46de12ad344-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:11:38.667585 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.667512    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhdbf\" (UniqueName: \"kubernetes.io/projected/d1b90694-e22f-47fe-b2b0-d46de12ad344-kube-api-access-xhdbf\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:11:38.667929 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.667911    2561 scope.go:117] "RemoveContainer" containerID="a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7"
Jul 20 11:11:38.675346 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.675324    2561 scope.go:117] "RemoveContainer" containerID="4678ea9bfd3c1daefc985dd1996d6ddd348f0d0d51ef239f523f0fee0ea12a7e"
Jul 20 11:11:38.681373 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.681352    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"]
Jul 20 11:11:38.683409 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.683382    2561 scope.go:117] "RemoveContainer" containerID="a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc"
Jul 20 11:11:38.683669 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:11:38.683645    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc\": container with ID starting with a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc not found: ID does not exist" containerID="a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc"
Jul 20 11:11:38.683718 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.683680    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc"} err="failed to get container status \"a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc\": rpc error: code = NotFound desc = could not find container \"a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc\": container with ID starting with a940da436472b0649f697f21dbd4b6fe1e4251472f9b74e60a1178710db415dc not found: ID does not exist"
Jul 20 11:11:38.683718 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.683703    2561 scope.go:117] "RemoveContainer" containerID="a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7"
Jul 20 11:11:38.683971 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:11:38.683952    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7\": container with ID starting with a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7 not found: ID does not exist" containerID="a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7"
Jul 20 11:11:38.684022 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.683978    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7"} err="failed to get container status \"a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7\": rpc error: code = NotFound desc = could not find container \"a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7\": container with ID starting with a293aefb077bb0d7a3624e750ef09066e3647a811a37bef4d3eb8d1fd94e5ae7 not found: ID does not exist"
Jul 20 11:11:38.684022 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.683999    2561 scope.go:117] "RemoveContainer" containerID="4678ea9bfd3c1daefc985dd1996d6ddd348f0d0d51ef239f523f0fee0ea12a7e"
Jul 20 11:11:38.684279 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:11:38.684258    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4678ea9bfd3c1daefc985dd1996d6ddd348f0d0d51ef239f523f0fee0ea12a7e\": container with ID starting with 4678ea9bfd3c1daefc985dd1996d6ddd348f0d0d51ef239f523f0fee0ea12a7e not found: ID does not exist" containerID="4678ea9bfd3c1daefc985dd1996d6ddd348f0d0d51ef239f523f0fee0ea12a7e"
Jul 20 11:11:38.684338 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.684285    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4678ea9bfd3c1daefc985dd1996d6ddd348f0d0d51ef239f523f0fee0ea12a7e"} err="failed to get container status \"4678ea9bfd3c1daefc985dd1996d6ddd348f0d0d51ef239f523f0fee0ea12a7e\": rpc error: code = NotFound desc = could not find container \"4678ea9bfd3c1daefc985dd1996d6ddd348f0d0d51ef239f523f0fee0ea12a7e\": container with ID starting with 4678ea9bfd3c1daefc985dd1996d6ddd348f0d0d51ef239f523f0fee0ea12a7e not found: ID does not exist"
Jul 20 11:11:38.684917 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.684902    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5"]
Jul 20 11:11:38.764970 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:38.764944    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" path="/var/lib/kubelet/pods/d1b90694-e22f-47fe-b2b0-d46de12ad344/volumes"
Jul 20 11:11:39.434411 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:39.434351    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nvdm5" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.65:8643/healthz\": context deadline exceeded"
Jul 20 11:11:41.672285 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:41.672250    2561 generic.go:356] "Generic (PLEG): container finished" podID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerID="928c253bca6c278fd08d3f1db70fc5fabd1b63aff4af8a08de991e384161cf51" exitCode=0
Jul 20 11:11:41.672667 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:41.672324    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" event={"ID":"edeba85e-0ab9-47da-aa7b-a8f228c969bd","Type":"ContainerDied","Data":"928c253bca6c278fd08d3f1db70fc5fabd1b63aff4af8a08de991e384161cf51"}
Jul 20 11:11:42.678639 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:42.678606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" event={"ID":"edeba85e-0ab9-47da-aa7b-a8f228c969bd","Type":"ContainerStarted","Data":"8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922"}
Jul 20 11:11:42.679017 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:42.678668    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" event={"ID":"edeba85e-0ab9-47da-aa7b-a8f228c969bd","Type":"ContainerStarted","Data":"0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416"}
Jul 20 11:11:42.679017 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:42.678929    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:42.679099 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:42.679078    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:42.680377 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:42.680347    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Jul 20 11:11:42.698826 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:42.698781    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podStartSLOduration=3.077220674 podStartE2EDuration="6.69876911s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.621548436s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:11:42.696932563 +0000 UTC m=+4482.535031161" watchObservedRunningTime="2026-07-20 11:11:42.69876911 +0000 UTC m=+4482.536867695"
Jul 20 11:11:43.682928 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:43.682886    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Jul 20 11:11:48.687364 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:48.687330    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:11:48.687913 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:48.687885    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Jul 20 11:11:58.688498 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:11:58.688416    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Jul 20 11:12:08.687861 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:08.687811    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Jul 20 11:12:16.255422 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:16.255394    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 11:12:16.263335 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:16.263313    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 11:12:18.688550 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:18.688513    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Jul 20 11:12:28.688765 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:28.688723    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Jul 20 11:12:38.688052 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:38.688002    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Jul 20 11:12:48.688896 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:48.688867    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:12:56.857195 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:56.857162    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"]
Jul 20 11:12:56.857700 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:56.857405    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container" containerID="cri-o://0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416" gracePeriod=30
Jul 20 11:12:56.857700 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:56.857437    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kube-rbac-proxy" containerID="cri-o://8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922" gracePeriod=30
Jul 20 11:12:57.956093 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:57.956063    2561 generic.go:356] "Generic (PLEG): container finished" podID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerID="8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922" exitCode=2
Jul 20 11:12:57.956450 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:57.956137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" event={"ID":"edeba85e-0ab9-47da-aa7b-a8f228c969bd","Type":"ContainerDied","Data":"8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922"}
Jul 20 11:12:58.683213 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:58.683172    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.66:8643/healthz\": dial tcp 10.132.0.66:8643: connect: connection refused"
Jul 20 11:12:58.688496 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:12:58.688464    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Jul 20 11:13:00.313427 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.313394    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:13:00.353337 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.353311    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kserve-provision-location\") pod \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") "
Jul 20 11:13:00.353480 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.353346    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kube-rbac-proxy-sar-config\") pod \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") "
Jul 20 11:13:00.353480 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.353368    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edeba85e-0ab9-47da-aa7b-a8f228c969bd-cabundle-cert\") pod \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") "
Jul 20 11:13:00.353480 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.353416    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edeba85e-0ab9-47da-aa7b-a8f228c969bd-proxy-tls\") pod \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") "
Jul 20 11:13:00.353480 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.353439    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhdc4\" (UniqueName: \"kubernetes.io/projected/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kube-api-access-rhdc4\") pod \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\" (UID: \"edeba85e-0ab9-47da-aa7b-a8f228c969bd\") "
Jul 20 11:13:00.353714 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.353687    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "edeba85e-0ab9-47da-aa7b-a8f228c969bd" (UID: "edeba85e-0ab9-47da-aa7b-a8f228c969bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:13:00.353714 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.353686    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "edeba85e-0ab9-47da-aa7b-a8f228c969bd" (UID: "edeba85e-0ab9-47da-aa7b-a8f228c969bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:13:00.353825 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.353740    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edeba85e-0ab9-47da-aa7b-a8f228c969bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "edeba85e-0ab9-47da-aa7b-a8f228c969bd" (UID: "edeba85e-0ab9-47da-aa7b-a8f228c969bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:13:00.355624 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.355597    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kube-api-access-rhdc4" (OuterVolumeSpecName: "kube-api-access-rhdc4") pod "edeba85e-0ab9-47da-aa7b-a8f228c969bd" (UID: "edeba85e-0ab9-47da-aa7b-a8f228c969bd"). InnerVolumeSpecName "kube-api-access-rhdc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:13:00.355700 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.355629    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edeba85e-0ab9-47da-aa7b-a8f228c969bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "edeba85e-0ab9-47da-aa7b-a8f228c969bd" (UID: "edeba85e-0ab9-47da-aa7b-a8f228c969bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:13:00.454231 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.454147    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edeba85e-0ab9-47da-aa7b-a8f228c969bd-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:13:00.454231 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.454176    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edeba85e-0ab9-47da-aa7b-a8f228c969bd-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:13:00.454231 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.454185    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhdc4\" (UniqueName: \"kubernetes.io/projected/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kube-api-access-rhdc4\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:13:00.454231 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.454195    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:13:00.454231 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.454204    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edeba85e-0ab9-47da-aa7b-a8f228c969bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:13:00.968606 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.968575    2561 generic.go:356] "Generic (PLEG): container finished" podID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerID="0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416" exitCode=0
Jul 20 11:13:00.968759 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.968627    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"
Jul 20 11:13:00.968759 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.968641    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" event={"ID":"edeba85e-0ab9-47da-aa7b-a8f228c969bd","Type":"ContainerDied","Data":"0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416"}
Jul 20 11:13:00.968759 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.968695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz" event={"ID":"edeba85e-0ab9-47da-aa7b-a8f228c969bd","Type":"ContainerDied","Data":"2340fe59c458aba0fcb69bd66316f2ce033feb6d1275d878bd82031cc6d1e9de"}
Jul 20 11:13:00.968759 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.968723    2561 scope.go:117] "RemoveContainer" containerID="8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922"
Jul 20 11:13:00.976263 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.976248    2561 scope.go:117] "RemoveContainer" containerID="0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416"
Jul 20 11:13:00.982997 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.982982    2561 scope.go:117] "RemoveContainer" containerID="928c253bca6c278fd08d3f1db70fc5fabd1b63aff4af8a08de991e384161cf51"
Jul 20 11:13:00.987299 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.987274    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"]
Jul 20 11:13:00.990951 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.990929    2561 scope.go:117] "RemoveContainer" containerID="8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922"
Jul 20 11:13:00.991078 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.991060    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w7rwz"]
Jul 20 11:13:00.991264 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:13:00.991233    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922\": container with ID starting with 8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922 not found: ID does not exist" containerID="8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922"
Jul 20 11:13:00.991374 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.991271    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922"} err="failed to get container status \"8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922\": rpc error: code = NotFound desc = could not find container \"8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922\": container with ID starting with 8144411dec5cce5017f4a5f409a8800ee5fde062ab9042ecf06a8c842fcf9922 not found: ID does not exist"
Jul 20 11:13:00.991374 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.991290    2561 scope.go:117] "RemoveContainer" containerID="0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416"
Jul 20 11:13:00.991530 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:13:00.991513    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416\": container with ID starting with 0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416 not found: ID does not exist" containerID="0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416"
Jul 20 11:13:00.991595 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.991541    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416"} err="failed to get container status \"0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416\": rpc error: code = NotFound desc = could not find container \"0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416\": container with ID starting with 0018dfcdf278e91d62fcab03835ff9eec42d0d81e7584eee1ba7b5a8ef62e416 not found: ID does not exist"
Jul 20 11:13:00.991595 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.991563    2561 scope.go:117] "RemoveContainer" containerID="928c253bca6c278fd08d3f1db70fc5fabd1b63aff4af8a08de991e384161cf51"
Jul 20 11:13:00.991810 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:13:00.991792    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928c253bca6c278fd08d3f1db70fc5fabd1b63aff4af8a08de991e384161cf51\": container with ID starting with 928c253bca6c278fd08d3f1db70fc5fabd1b63aff4af8a08de991e384161cf51 not found: ID does not exist" containerID="928c253bca6c278fd08d3f1db70fc5fabd1b63aff4af8a08de991e384161cf51"
Jul 20 11:13:00.991874 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:00.991814    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928c253bca6c278fd08d3f1db70fc5fabd1b63aff4af8a08de991e384161cf51"} err="failed to get container status \"928c253bca6c278fd08d3f1db70fc5fabd1b63aff4af8a08de991e384161cf51\": rpc error: code = NotFound desc = could not find container \"928c253bca6c278fd08d3f1db70fc5fabd1b63aff4af8a08de991e384161cf51\": container with ID starting with 928c253bca6c278fd08d3f1db70fc5fabd1b63aff4af8a08de991e384161cf51 not found: ID does not exist"
Jul 20 11:13:02.012819 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.012787    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"]
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013026    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="kserve-container"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013038    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="kserve-container"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013050    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013056    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013063    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="kube-rbac-proxy"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013068    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="kube-rbac-proxy"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013079    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kube-rbac-proxy"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013084    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kube-rbac-proxy"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013092    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="storage-initializer"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013097    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="storage-initializer"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013103    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="storage-initializer"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013108    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="storage-initializer"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013160    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kserve-container"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013168    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="kserve-container"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013174    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1b90694-e22f-47fe-b2b0-d46de12ad344" containerName="kube-rbac-proxy"
Jul 20 11:13:02.013206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.013181    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" containerName="kube-rbac-proxy"
Jul 20 11:13:02.017463 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.017443    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.020004 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.019985    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 20 11:13:02.020004 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.020000    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 11:13:02.020214 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.020050    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pkbv\""
Jul 20 11:13:02.020214 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.020082    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 11:13:02.020214 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.020106    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 20 11:13:02.021055 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.021040    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 20 11:13:02.026699 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.026673    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"]
Jul 20 11:13:02.068700 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.068665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e478804-4bca-42c9-ac80-0b834babef9d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5df5f6df69-rdm2f\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.068860 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.068727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjsb\" (UniqueName: \"kubernetes.io/projected/0e478804-4bca-42c9-ac80-0b834babef9d-kube-api-access-rmjsb\") pod \"isvc-sklearn-s3-predictor-5df5f6df69-rdm2f\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.068860 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.068768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e478804-4bca-42c9-ac80-0b834babef9d-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5df5f6df69-rdm2f\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.068860 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.068802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e478804-4bca-42c9-ac80-0b834babef9d-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5df5f6df69-rdm2f\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.169576 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.169543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e478804-4bca-42c9-ac80-0b834babef9d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5df5f6df69-rdm2f\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.169754 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.169599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjsb\" (UniqueName: \"kubernetes.io/projected/0e478804-4bca-42c9-ac80-0b834babef9d-kube-api-access-rmjsb\") pod \"isvc-sklearn-s3-predictor-5df5f6df69-rdm2f\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.169754 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.169640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e478804-4bca-42c9-ac80-0b834babef9d-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5df5f6df69-rdm2f\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.169754 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.169672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e478804-4bca-42c9-ac80-0b834babef9d-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5df5f6df69-rdm2f\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.170150 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.170131    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e478804-4bca-42c9-ac80-0b834babef9d-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5df5f6df69-rdm2f\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.170250 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.170233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e478804-4bca-42c9-ac80-0b834babef9d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5df5f6df69-rdm2f\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.172140 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.172102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e478804-4bca-42c9-ac80-0b834babef9d-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5df5f6df69-rdm2f\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.177272 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.177252    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjsb\" (UniqueName: \"kubernetes.io/projected/0e478804-4bca-42c9-ac80-0b834babef9d-kube-api-access-rmjsb\") pod \"isvc-sklearn-s3-predictor-5df5f6df69-rdm2f\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.329135 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.329015    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:02.455193 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.455166    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"]
Jul 20 11:13:02.457535 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:13:02.457507    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e478804_4bca_42c9_ac80_0b834babef9d.slice/crio-f0d7fa68ac9999401c35d034ff7e81fa674c45cd3979d51a58f9e0cd518ae922 WatchSource:0}: Error finding container f0d7fa68ac9999401c35d034ff7e81fa674c45cd3979d51a58f9e0cd518ae922: Status 404 returned error can't find the container with id f0d7fa68ac9999401c35d034ff7e81fa674c45cd3979d51a58f9e0cd518ae922
Jul 20 11:13:02.766993 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.766963    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edeba85e-0ab9-47da-aa7b-a8f228c969bd" path="/var/lib/kubelet/pods/edeba85e-0ab9-47da-aa7b-a8f228c969bd/volumes"
Jul 20 11:13:02.979531 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.979498    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" event={"ID":"0e478804-4bca-42c9-ac80-0b834babef9d","Type":"ContainerStarted","Data":"b36588a1b46b6e6363c3058630d521bd157ab6923d652447d20d7c00c5c9797c"}
Jul 20 11:13:02.979699 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:02.979555    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" event={"ID":"0e478804-4bca-42c9-ac80-0b834babef9d","Type":"ContainerStarted","Data":"f0d7fa68ac9999401c35d034ff7e81fa674c45cd3979d51a58f9e0cd518ae922"}
Jul 20 11:13:03.984741 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:03.984709    2561 generic.go:356] "Generic (PLEG): container finished" podID="0e478804-4bca-42c9-ac80-0b834babef9d" containerID="b36588a1b46b6e6363c3058630d521bd157ab6923d652447d20d7c00c5c9797c" exitCode=0
Jul 20 11:13:03.985207 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:03.984797    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" event={"ID":"0e478804-4bca-42c9-ac80-0b834babef9d","Type":"ContainerDied","Data":"b36588a1b46b6e6363c3058630d521bd157ab6923d652447d20d7c00c5c9797c"}
Jul 20 11:13:04.990958 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:04.990926    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" event={"ID":"0e478804-4bca-42c9-ac80-0b834babef9d","Type":"ContainerStarted","Data":"abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe"}
Jul 20 11:13:04.991392 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:04.990980    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" event={"ID":"0e478804-4bca-42c9-ac80-0b834babef9d","Type":"ContainerStarted","Data":"ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e"}
Jul 20 11:13:04.991392 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:04.991171    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:04.991392 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:04.991229    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:04.992436 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:04.992410    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Jul 20 11:13:05.009328 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:05.009290    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podStartSLOduration=3.017116348 podStartE2EDuration="4.009277144s" totalImagesPullingTime="0s" totalInitContainerRuntime="992.160796ms" isStatefulPod=false podCreationTimestamp="2026-07-20 11:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:13:05.00822411 +0000 UTC m=+4564.846322711" watchObservedRunningTime="2026-07-20 11:13:05.009277144 +0000 UTC m=+4564.847375707"
Jul 20 11:13:05.997204 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:05.997159    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Jul 20 11:13:11.001621 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:11.001590    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:13:11.002188 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:11.002157    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Jul 20 11:13:21.002335 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:21.002295    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Jul 20 11:13:31.002325 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:31.002289    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Jul 20 11:13:41.002193 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:41.002153    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Jul 20 11:13:51.002253 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:13:51.002216    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Jul 20 11:14:01.002199 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:01.002160    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Jul 20 11:14:11.002868 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:11.002830    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Jul 20 11:14:12.761258 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:12.761220    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Jul 20 11:14:22.764694 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:22.764665    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:14:32.161281 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.161249    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"]
Jul 20 11:14:32.161745 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.161464    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container" containerID="cri-o://ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e" gracePeriod=30
Jul 20 11:14:32.161745 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.161543    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kube-rbac-proxy" containerID="cri-o://abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe" gracePeriod=30
Jul 20 11:14:32.302363 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.302323    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"]
Jul 20 11:14:32.306152 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.306130    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.308484 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.308459    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 20 11:14:32.308641 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.308600    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 11:14:32.308930 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.308909    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 20 11:14:32.312273 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.312250    2561 generic.go:356] "Generic (PLEG): container finished" podID="0e478804-4bca-42c9-ac80-0b834babef9d" containerID="abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe" exitCode=2
Jul 20 11:14:32.312360 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.312322    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" event={"ID":"0e478804-4bca-42c9-ac80-0b834babef9d","Type":"ContainerDied","Data":"abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe"}
Jul 20 11:14:32.315842 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.315805    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"]
Jul 20 11:14:32.332627 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.332599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/024815a1-6f9f-45f1-a9f7-334d97c16cc0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.332745 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.332638    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.332745 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.332685    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.332745 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.332716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/024815a1-6f9f-45f1-a9f7-334d97c16cc0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.332911 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.332773    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwgh2\" (UniqueName: \"kubernetes.io/projected/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kube-api-access-pwgh2\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.433847 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.433764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.433847 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.433810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.433847 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.433845    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/024815a1-6f9f-45f1-a9f7-334d97c16cc0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.434177 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.433981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwgh2\" (UniqueName: \"kubernetes.io/projected/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kube-api-access-pwgh2\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.434177 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.434040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/024815a1-6f9f-45f1-a9f7-334d97c16cc0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.434310 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.434236    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.434569 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.434546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/024815a1-6f9f-45f1-a9f7-334d97c16cc0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.434652 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.434567    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.436499 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.436476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/024815a1-6f9f-45f1-a9f7-334d97c16cc0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.441366 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.441340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwgh2\" (UniqueName: \"kubernetes.io/projected/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kube-api-access-pwgh2\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.617251 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.617216    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:32.750394 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.750368    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"]
Jul 20 11:14:32.752837 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:14:32.752814    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024815a1_6f9f_45f1_a9f7_334d97c16cc0.slice/crio-cbf72637071aa322cef7a7841909d058d1af9c7af441b9a9732066af90d74e63 WatchSource:0}: Error finding container cbf72637071aa322cef7a7841909d058d1af9c7af441b9a9732066af90d74e63: Status 404 returned error can't find the container with id cbf72637071aa322cef7a7841909d058d1af9c7af441b9a9732066af90d74e63
Jul 20 11:14:32.768150 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:32.762380    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Jul 20 11:14:33.318432 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:33.318386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" event={"ID":"024815a1-6f9f-45f1-a9f7-334d97c16cc0","Type":"ContainerStarted","Data":"6e1d888e8ecd538dbcac95da464212d0ef19b481dde3037bce044c2fcc9c987b"}
Jul 20 11:14:33.318802 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:33.318476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" event={"ID":"024815a1-6f9f-45f1-a9f7-334d97c16cc0","Type":"ContainerStarted","Data":"cbf72637071aa322cef7a7841909d058d1af9c7af441b9a9732066af90d74e63"}
Jul 20 11:14:34.323444 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:34.323411    2561 generic.go:356] "Generic (PLEG): container finished" podID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerID="6e1d888e8ecd538dbcac95da464212d0ef19b481dde3037bce044c2fcc9c987b" exitCode=0
Jul 20 11:14:34.323854 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:34.323469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" event={"ID":"024815a1-6f9f-45f1-a9f7-334d97c16cc0","Type":"ContainerDied","Data":"6e1d888e8ecd538dbcac95da464212d0ef19b481dde3037bce044c2fcc9c987b"}
Jul 20 11:14:35.329047 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:35.329013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" event={"ID":"024815a1-6f9f-45f1-a9f7-334d97c16cc0","Type":"ContainerStarted","Data":"e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642"}
Jul 20 11:14:35.329535 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:35.329075    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" event={"ID":"024815a1-6f9f-45f1-a9f7-334d97c16cc0","Type":"ContainerStarted","Data":"95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030"}
Jul 20 11:14:35.329535 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:35.329134    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:35.329535 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:35.329170    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:35.330542 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:35.330516    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Jul 20 11:14:35.348883 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:35.348837    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podStartSLOduration=2.415431883 podStartE2EDuration="3.348823685s" totalImagesPullingTime="0s" totalInitContainerRuntime="933.391802ms" isStatefulPod=false podCreationTimestamp="2026-07-20 11:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:14:35.347032535 +0000 UTC m=+4655.185131137" watchObservedRunningTime="2026-07-20 11:14:35.348823685 +0000 UTC m=+4655.186922270"
Jul 20 11:14:35.998325 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:35.998277    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused"
Jul 20 11:14:36.332737 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.332640    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Jul 20 11:14:36.705335 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.705311    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:14:36.766049 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.766012    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e478804-4bca-42c9-ac80-0b834babef9d-proxy-tls\") pod \"0e478804-4bca-42c9-ac80-0b834babef9d\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") "
Jul 20 11:14:36.766236 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.766062    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmjsb\" (UniqueName: \"kubernetes.io/projected/0e478804-4bca-42c9-ac80-0b834babef9d-kube-api-access-rmjsb\") pod \"0e478804-4bca-42c9-ac80-0b834babef9d\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") "
Jul 20 11:14:36.766236 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.766162    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e478804-4bca-42c9-ac80-0b834babef9d-kube-rbac-proxy-sar-config\") pod \"0e478804-4bca-42c9-ac80-0b834babef9d\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") "
Jul 20 11:14:36.766236 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.766188    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e478804-4bca-42c9-ac80-0b834babef9d-kserve-provision-location\") pod \"0e478804-4bca-42c9-ac80-0b834babef9d\" (UID: \"0e478804-4bca-42c9-ac80-0b834babef9d\") "
Jul 20 11:14:36.766545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.766515    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e478804-4bca-42c9-ac80-0b834babef9d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e478804-4bca-42c9-ac80-0b834babef9d" (UID: "0e478804-4bca-42c9-ac80-0b834babef9d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:14:36.766614 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.766530    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e478804-4bca-42c9-ac80-0b834babef9d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e478804-4bca-42c9-ac80-0b834babef9d" (UID: "0e478804-4bca-42c9-ac80-0b834babef9d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:14:36.768126 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.768088    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e478804-4bca-42c9-ac80-0b834babef9d-kube-api-access-rmjsb" (OuterVolumeSpecName: "kube-api-access-rmjsb") pod "0e478804-4bca-42c9-ac80-0b834babef9d" (UID: "0e478804-4bca-42c9-ac80-0b834babef9d"). InnerVolumeSpecName "kube-api-access-rmjsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:14:36.768338 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.768322    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e478804-4bca-42c9-ac80-0b834babef9d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e478804-4bca-42c9-ac80-0b834babef9d" (UID: "0e478804-4bca-42c9-ac80-0b834babef9d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:14:36.866949 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.866919    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e478804-4bca-42c9-ac80-0b834babef9d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:14:36.866949 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.866945    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e478804-4bca-42c9-ac80-0b834babef9d-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:14:36.866949 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.866955    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e478804-4bca-42c9-ac80-0b834babef9d-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:14:36.867193 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:36.866966    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmjsb\" (UniqueName: \"kubernetes.io/projected/0e478804-4bca-42c9-ac80-0b834babef9d-kube-api-access-rmjsb\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:14:37.338324 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.338287    2561 generic.go:356] "Generic (PLEG): container finished" podID="0e478804-4bca-42c9-ac80-0b834babef9d" containerID="ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e" exitCode=0
Jul 20 11:14:37.338716 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.338332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" event={"ID":"0e478804-4bca-42c9-ac80-0b834babef9d","Type":"ContainerDied","Data":"ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e"}
Jul 20 11:14:37.338716 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.338352    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"
Jul 20 11:14:37.338716 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.338388    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f" event={"ID":"0e478804-4bca-42c9-ac80-0b834babef9d","Type":"ContainerDied","Data":"f0d7fa68ac9999401c35d034ff7e81fa674c45cd3979d51a58f9e0cd518ae922"}
Jul 20 11:14:37.338716 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.338416    2561 scope.go:117] "RemoveContainer" containerID="abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe"
Jul 20 11:14:37.348175 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.348157    2561 scope.go:117] "RemoveContainer" containerID="ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e"
Jul 20 11:14:37.355275 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.355246    2561 scope.go:117] "RemoveContainer" containerID="b36588a1b46b6e6363c3058630d521bd157ab6923d652447d20d7c00c5c9797c"
Jul 20 11:14:37.362738 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.362719    2561 scope.go:117] "RemoveContainer" containerID="abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe"
Jul 20 11:14:37.363006 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.362988    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"]
Jul 20 11:14:37.363059 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:14:37.363007    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe\": container with ID starting with abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe not found: ID does not exist" containerID="abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe"
Jul 20 11:14:37.363059 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.363029    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe"} err="failed to get container status \"abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe\": rpc error: code = NotFound desc = could not find container \"abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe\": container with ID starting with abce56158ddf8042a648488e6f26a12d156303b8df17c7c3fd379a007e0cd4fe not found: ID does not exist"
Jul 20 11:14:37.363059 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.363046    2561 scope.go:117] "RemoveContainer" containerID="ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e"
Jul 20 11:14:37.363309 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:14:37.363290    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e\": container with ID starting with ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e not found: ID does not exist" containerID="ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e"
Jul 20 11:14:37.363375 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.363318    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e"} err="failed to get container status \"ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e\": rpc error: code = NotFound desc = could not find container \"ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e\": container with ID starting with ce69f54de22b2c6926d60b26c2e30de6feb42232f5220ac858f857caf3eaa26e not found: ID does not exist"
Jul 20 11:14:37.363375 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.363341    2561 scope.go:117] "RemoveContainer" containerID="b36588a1b46b6e6363c3058630d521bd157ab6923d652447d20d7c00c5c9797c"
Jul 20 11:14:37.363585 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:14:37.363565    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36588a1b46b6e6363c3058630d521bd157ab6923d652447d20d7c00c5c9797c\": container with ID starting with b36588a1b46b6e6363c3058630d521bd157ab6923d652447d20d7c00c5c9797c not found: ID does not exist" containerID="b36588a1b46b6e6363c3058630d521bd157ab6923d652447d20d7c00c5c9797c"
Jul 20 11:14:37.363681 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.363585    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36588a1b46b6e6363c3058630d521bd157ab6923d652447d20d7c00c5c9797c"} err="failed to get container status \"b36588a1b46b6e6363c3058630d521bd157ab6923d652447d20d7c00c5c9797c\": rpc error: code = NotFound desc = could not find container \"b36588a1b46b6e6363c3058630d521bd157ab6923d652447d20d7c00c5c9797c\": container with ID starting with b36588a1b46b6e6363c3058630d521bd157ab6923d652447d20d7c00c5c9797c not found: ID does not exist"
Jul 20 11:14:37.366180 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:37.366151    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5df5f6df69-rdm2f"]
Jul 20 11:14:38.765719 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:38.765688    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" path="/var/lib/kubelet/pods/0e478804-4bca-42c9-ac80-0b834babef9d/volumes"
Jul 20 11:14:41.337317 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:41.337289    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:14:41.337744 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:41.337719    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Jul 20 11:14:51.338584 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:14:51.338545    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Jul 20 11:15:01.337735 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:01.337696    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Jul 20 11:15:11.338202 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:11.338163    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Jul 20 11:15:21.338214 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:21.338172    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Jul 20 11:15:31.338610 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:31.338569    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Jul 20 11:15:41.338277 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:41.338237    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Jul 20 11:15:51.339134 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:51.339077    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:15:52.332762 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:52.332731    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"]
Jul 20 11:15:52.333018 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:52.332989    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container" containerID="cri-o://95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030" gracePeriod=30
Jul 20 11:15:52.333092 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:52.333038    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kube-rbac-proxy" containerID="cri-o://e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642" gracePeriod=30
Jul 20 11:15:52.617093 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:52.617066    2561 generic.go:356] "Generic (PLEG): container finished" podID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerID="e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642" exitCode=2
Jul 20 11:15:52.617493 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:52.617137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" event={"ID":"024815a1-6f9f-45f1-a9f7-334d97c16cc0","Type":"ContainerDied","Data":"e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642"}
Jul 20 11:15:53.387036 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.387007    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"]
Jul 20 11:15:53.387302 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.387289    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="storage-initializer"
Jul 20 11:15:53.387351 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.387304    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="storage-initializer"
Jul 20 11:15:53.387351 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.387312    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kube-rbac-proxy"
Jul 20 11:15:53.387351 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.387318    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kube-rbac-proxy"
Jul 20 11:15:53.387351 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.387330    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container"
Jul 20 11:15:53.387351 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.387335    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container"
Jul 20 11:15:53.387536 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.387373    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kube-rbac-proxy"
Jul 20 11:15:53.387536 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.387382    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e478804-4bca-42c9-ac80-0b834babef9d" containerName="kserve-container"
Jul 20 11:15:53.390062 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.390044    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:53.392298 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.392270    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 20 11:15:53.392432 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.392301    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 20 11:15:53.401021 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.400997    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"]
Jul 20 11:15:53.433531 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.433500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98sw\" (UniqueName: \"kubernetes.io/projected/4dc732f0-5319-4314-8189-9bb0ba826b4d-kube-api-access-h98sw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:53.433692 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.433541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dc732f0-5319-4314-8189-9bb0ba826b4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:53.433692 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.433566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dc732f0-5319-4314-8189-9bb0ba826b4d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:53.433801 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.433682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dc732f0-5319-4314-8189-9bb0ba826b4d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:53.534737 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.534701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98sw\" (UniqueName: \"kubernetes.io/projected/4dc732f0-5319-4314-8189-9bb0ba826b4d-kube-api-access-h98sw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:53.534920 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.534746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dc732f0-5319-4314-8189-9bb0ba826b4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:53.534920 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.534776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dc732f0-5319-4314-8189-9bb0ba826b4d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:53.534920 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.534824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dc732f0-5319-4314-8189-9bb0ba826b4d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:53.535103 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:15:53.534954    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 20 11:15:53.535103 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:15:53.535045    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dc732f0-5319-4314-8189-9bb0ba826b4d-proxy-tls podName:4dc732f0-5319-4314-8189-9bb0ba826b4d nodeName:}" failed. No retries permitted until 2026-07-20 11:15:54.035020008 +0000 UTC m=+4733.873118574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4dc732f0-5319-4314-8189-9bb0ba826b4d-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p" (UID: "4dc732f0-5319-4314-8189-9bb0ba826b4d") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 20 11:15:53.535336 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.535313    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dc732f0-5319-4314-8189-9bb0ba826b4d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:53.535442 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.535423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dc732f0-5319-4314-8189-9bb0ba826b4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:53.543056 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:53.543033    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98sw\" (UniqueName: \"kubernetes.io/projected/4dc732f0-5319-4314-8189-9bb0ba826b4d-kube-api-access-h98sw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:54.039664 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:54.039631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dc732f0-5319-4314-8189-9bb0ba826b4d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:54.041950 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:54.041931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dc732f0-5319-4314-8189-9bb0ba826b4d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:54.299714 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:54.299620    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:15:54.428647 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:54.428624    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"]
Jul 20 11:15:54.431050 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:15:54.431021    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc732f0_5319_4314_8189_9bb0ba826b4d.slice/crio-9fb5baa007f811f9252c420ea666f1d205d9ecbdc4ea72f5ed3b9db992b554f7 WatchSource:0}: Error finding container 9fb5baa007f811f9252c420ea666f1d205d9ecbdc4ea72f5ed3b9db992b554f7: Status 404 returned error can't find the container with id 9fb5baa007f811f9252c420ea666f1d205d9ecbdc4ea72f5ed3b9db992b554f7
Jul 20 11:15:54.626049 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:54.626013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p" event={"ID":"4dc732f0-5319-4314-8189-9bb0ba826b4d","Type":"ContainerStarted","Data":"4d85bc563be42ca9fcb4e2d069e1eec96c830cdc30ea8dd0ef69fdc7042c5f3d"}
Jul 20 11:15:54.626270 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:54.626073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p" event={"ID":"4dc732f0-5319-4314-8189-9bb0ba826b4d","Type":"ContainerStarted","Data":"9fb5baa007f811f9252c420ea666f1d205d9ecbdc4ea72f5ed3b9db992b554f7"}
Jul 20 11:15:56.332885 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:56.332841    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.68:8643/healthz\": dial tcp 10.132.0.68:8643: connect: connection refused"
Jul 20 11:15:56.975782 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:56.975752    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:15:57.063025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.062947    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kserve-provision-location\") pod \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") "
Jul 20 11:15:57.063025 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.063010    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/024815a1-6f9f-45f1-a9f7-334d97c16cc0-cabundle-cert\") pod \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") "
Jul 20 11:15:57.063231 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.063040    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwgh2\" (UniqueName: \"kubernetes.io/projected/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kube-api-access-pwgh2\") pod \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") "
Jul 20 11:15:57.063231 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.063102    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/024815a1-6f9f-45f1-a9f7-334d97c16cc0-proxy-tls\") pod \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") "
Jul 20 11:15:57.063231 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.063194    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kube-rbac-proxy-sar-config\") pod \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\" (UID: \"024815a1-6f9f-45f1-a9f7-334d97c16cc0\") "
Jul 20 11:15:57.063378 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.063351    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "024815a1-6f9f-45f1-a9f7-334d97c16cc0" (UID: "024815a1-6f9f-45f1-a9f7-334d97c16cc0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:15:57.063510 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.063490    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024815a1-6f9f-45f1-a9f7-334d97c16cc0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "024815a1-6f9f-45f1-a9f7-334d97c16cc0" (UID: "024815a1-6f9f-45f1-a9f7-334d97c16cc0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:15:57.063566 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.063511    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "024815a1-6f9f-45f1-a9f7-334d97c16cc0" (UID: "024815a1-6f9f-45f1-a9f7-334d97c16cc0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:15:57.065207 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.065183    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kube-api-access-pwgh2" (OuterVolumeSpecName: "kube-api-access-pwgh2") pod "024815a1-6f9f-45f1-a9f7-334d97c16cc0" (UID: "024815a1-6f9f-45f1-a9f7-334d97c16cc0"). InnerVolumeSpecName "kube-api-access-pwgh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:15:57.065311 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.065190    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024815a1-6f9f-45f1-a9f7-334d97c16cc0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "024815a1-6f9f-45f1-a9f7-334d97c16cc0" (UID: "024815a1-6f9f-45f1-a9f7-334d97c16cc0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:15:57.163852 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.163821    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:15:57.163852 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.163847    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:15:57.163852 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.163857    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/024815a1-6f9f-45f1-a9f7-334d97c16cc0-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:15:57.164073 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.163866    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwgh2\" (UniqueName: \"kubernetes.io/projected/024815a1-6f9f-45f1-a9f7-334d97c16cc0-kube-api-access-pwgh2\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:15:57.164073 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.163874    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/024815a1-6f9f-45f1-a9f7-334d97c16cc0-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:15:57.639058 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.639023    2561 generic.go:356] "Generic (PLEG): container finished" podID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerID="95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030" exitCode=0
Jul 20 11:15:57.639554 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.639081    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"
Jul 20 11:15:57.639554 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.639083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" event={"ID":"024815a1-6f9f-45f1-a9f7-334d97c16cc0","Type":"ContainerDied","Data":"95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030"}
Jul 20 11:15:57.639554 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.639164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86" event={"ID":"024815a1-6f9f-45f1-a9f7-334d97c16cc0","Type":"ContainerDied","Data":"cbf72637071aa322cef7a7841909d058d1af9c7af441b9a9732066af90d74e63"}
Jul 20 11:15:57.639554 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.639204    2561 scope.go:117] "RemoveContainer" containerID="e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642"
Jul 20 11:15:57.649692 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.649520    2561 scope.go:117] "RemoveContainer" containerID="95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030"
Jul 20 11:15:57.658081 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.658061    2561 scope.go:117] "RemoveContainer" containerID="6e1d888e8ecd538dbcac95da464212d0ef19b481dde3037bce044c2fcc9c987b"
Jul 20 11:15:57.663072 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.663046    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"]
Jul 20 11:15:57.665346 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.665319    2561 scope.go:117] "RemoveContainer" containerID="e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642"
Jul 20 11:15:57.665602 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:15:57.665576    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642\": container with ID starting with e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642 not found: ID does not exist" containerID="e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642"
Jul 20 11:15:57.665661 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.665605    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642"} err="failed to get container status \"e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642\": rpc error: code = NotFound desc = could not find container \"e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642\": container with ID starting with e8079d51c1ffde1305480d397d34f7fde6e433096b4e771ee638c8b1ae97b642 not found: ID does not exist"
Jul 20 11:15:57.665661 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.665623    2561 scope.go:117] "RemoveContainer" containerID="95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030"
Jul 20 11:15:57.665847 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:15:57.665831    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030\": container with ID starting with 95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030 not found: ID does not exist" containerID="95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030"
Jul 20 11:15:57.665889 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.665850    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030"} err="failed to get container status \"95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030\": rpc error: code = NotFound desc = could not find container \"95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030\": container with ID starting with 95756e8a3a45c2ee32f26ffc7d07b3911b11ff0ad1004712d25b84c9d6be6030 not found: ID does not exist"
Jul 20 11:15:57.665889 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.665862    2561 scope.go:117] "RemoveContainer" containerID="6e1d888e8ecd538dbcac95da464212d0ef19b481dde3037bce044c2fcc9c987b"
Jul 20 11:15:57.666093 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:15:57.666074    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e1d888e8ecd538dbcac95da464212d0ef19b481dde3037bce044c2fcc9c987b\": container with ID starting with 6e1d888e8ecd538dbcac95da464212d0ef19b481dde3037bce044c2fcc9c987b not found: ID does not exist" containerID="6e1d888e8ecd538dbcac95da464212d0ef19b481dde3037bce044c2fcc9c987b"
Jul 20 11:15:57.666176 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.666102    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e1d888e8ecd538dbcac95da464212d0ef19b481dde3037bce044c2fcc9c987b"} err="failed to get container status \"6e1d888e8ecd538dbcac95da464212d0ef19b481dde3037bce044c2fcc9c987b\": rpc error: code = NotFound desc = could not find container \"6e1d888e8ecd538dbcac95da464212d0ef19b481dde3037bce044c2fcc9c987b\": container with ID starting with 6e1d888e8ecd538dbcac95da464212d0ef19b481dde3037bce044c2fcc9c987b not found: ID does not exist"
Jul 20 11:15:57.668388 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:57.668368    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65b855989-29v86"]
Jul 20 11:15:58.644714 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:58.644686    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p_4dc732f0-5319-4314-8189-9bb0ba826b4d/storage-initializer/0.log"
Jul 20 11:15:58.645137 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:58.644722    2561 generic.go:356] "Generic (PLEG): container finished" podID="4dc732f0-5319-4314-8189-9bb0ba826b4d" containerID="4d85bc563be42ca9fcb4e2d069e1eec96c830cdc30ea8dd0ef69fdc7042c5f3d" exitCode=1
Jul 20 11:15:58.645137 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:58.644801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p" event={"ID":"4dc732f0-5319-4314-8189-9bb0ba826b4d","Type":"ContainerDied","Data":"4d85bc563be42ca9fcb4e2d069e1eec96c830cdc30ea8dd0ef69fdc7042c5f3d"}
Jul 20 11:15:58.765514 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:58.765485    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" path="/var/lib/kubelet/pods/024815a1-6f9f-45f1-a9f7-334d97c16cc0/volumes"
Jul 20 11:15:59.650492 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:59.650466    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p_4dc732f0-5319-4314-8189-9bb0ba826b4d/storage-initializer/0.log"
Jul 20 11:15:59.650862 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:15:59.650561    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p" event={"ID":"4dc732f0-5319-4314-8189-9bb0ba826b4d","Type":"ContainerStarted","Data":"de243c67f26cf194ff8b333969a66e42b49eb4118d9a8a96ee6ae420b11d8ecc"}
Jul 20 11:16:03.399488 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.399454    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"]
Jul 20 11:16:03.399996 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.399659    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p" podUID="4dc732f0-5319-4314-8189-9bb0ba826b4d" containerName="storage-initializer" containerID="cri-o://de243c67f26cf194ff8b333969a66e42b49eb4118d9a8a96ee6ae420b11d8ecc" gracePeriod=30
Jul 20 11:16:03.667509 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.667456    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p_4dc732f0-5319-4314-8189-9bb0ba826b4d/storage-initializer/1.log"
Jul 20 11:16:03.667940 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.667918    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p_4dc732f0-5319-4314-8189-9bb0ba826b4d/storage-initializer/0.log"
Jul 20 11:16:03.668052 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.667965    2561 generic.go:356] "Generic (PLEG): container finished" podID="4dc732f0-5319-4314-8189-9bb0ba826b4d" containerID="de243c67f26cf194ff8b333969a66e42b49eb4118d9a8a96ee6ae420b11d8ecc" exitCode=1
Jul 20 11:16:03.668105 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.668045    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p" event={"ID":"4dc732f0-5319-4314-8189-9bb0ba826b4d","Type":"ContainerDied","Data":"de243c67f26cf194ff8b333969a66e42b49eb4118d9a8a96ee6ae420b11d8ecc"}
Jul 20 11:16:03.668249 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.668169    2561 scope.go:117] "RemoveContainer" containerID="4d85bc563be42ca9fcb4e2d069e1eec96c830cdc30ea8dd0ef69fdc7042c5f3d"
Jul 20 11:16:03.740726 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.740705    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p_4dc732f0-5319-4314-8189-9bb0ba826b4d/storage-initializer/1.log"
Jul 20 11:16:03.744307 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.744289    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:16:03.806781 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.806760    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dc732f0-5319-4314-8189-9bb0ba826b4d-kserve-provision-location\") pod \"4dc732f0-5319-4314-8189-9bb0ba826b4d\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") "
Jul 20 11:16:03.806916 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.806788    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dc732f0-5319-4314-8189-9bb0ba826b4d-kube-rbac-proxy-sar-config\") pod \"4dc732f0-5319-4314-8189-9bb0ba826b4d\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") "
Jul 20 11:16:03.806916 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.806812    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dc732f0-5319-4314-8189-9bb0ba826b4d-proxy-tls\") pod \"4dc732f0-5319-4314-8189-9bb0ba826b4d\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") "
Jul 20 11:16:03.806916 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.806882    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h98sw\" (UniqueName: \"kubernetes.io/projected/4dc732f0-5319-4314-8189-9bb0ba826b4d-kube-api-access-h98sw\") pod \"4dc732f0-5319-4314-8189-9bb0ba826b4d\" (UID: \"4dc732f0-5319-4314-8189-9bb0ba826b4d\") "
Jul 20 11:16:03.807087 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.807039    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc732f0-5319-4314-8189-9bb0ba826b4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4dc732f0-5319-4314-8189-9bb0ba826b4d" (UID: "4dc732f0-5319-4314-8189-9bb0ba826b4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:16:03.807183 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.807095    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc732f0-5319-4314-8189-9bb0ba826b4d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4dc732f0-5319-4314-8189-9bb0ba826b4d" (UID: "4dc732f0-5319-4314-8189-9bb0ba826b4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:16:03.808771 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.808751    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc732f0-5319-4314-8189-9bb0ba826b4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4dc732f0-5319-4314-8189-9bb0ba826b4d" (UID: "4dc732f0-5319-4314-8189-9bb0ba826b4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:16:03.808898 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.808875    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc732f0-5319-4314-8189-9bb0ba826b4d-kube-api-access-h98sw" (OuterVolumeSpecName: "kube-api-access-h98sw") pod "4dc732f0-5319-4314-8189-9bb0ba826b4d" (UID: "4dc732f0-5319-4314-8189-9bb0ba826b4d"). InnerVolumeSpecName "kube-api-access-h98sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:16:03.908388 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.908346    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h98sw\" (UniqueName: \"kubernetes.io/projected/4dc732f0-5319-4314-8189-9bb0ba826b4d-kube-api-access-h98sw\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:16:03.908388 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.908386    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dc732f0-5319-4314-8189-9bb0ba826b4d-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:16:03.908388 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.908396    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dc732f0-5319-4314-8189-9bb0ba826b4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:16:03.908589 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:03.908406    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dc732f0-5319-4314-8189-9bb0ba826b4d-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:16:04.504140 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504093    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"]
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504370    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc732f0-5319-4314-8189-9bb0ba826b4d" containerName="storage-initializer"
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504382    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc732f0-5319-4314-8189-9bb0ba826b4d" containerName="storage-initializer"
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504390    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="storage-initializer"
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504396    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="storage-initializer"
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504404    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container"
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504410    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container"
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504417    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kube-rbac-proxy"
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504422    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kube-rbac-proxy"
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504432    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc732f0-5319-4314-8189-9bb0ba826b4d" containerName="storage-initializer"
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504438    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc732f0-5319-4314-8189-9bb0ba826b4d" containerName="storage-initializer"
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504480    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc732f0-5319-4314-8189-9bb0ba826b4d" containerName="storage-initializer"
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504489    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kserve-container"
Jul 20 11:16:04.504545 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504495    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="024815a1-6f9f-45f1-a9f7-334d97c16cc0" containerName="kube-rbac-proxy"
Jul 20 11:16:04.504951 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.504574    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc732f0-5319-4314-8189-9bb0ba826b4d" containerName="storage-initializer"
Jul 20 11:16:04.507666 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.507650    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.510186 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.510164    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 11:16:04.510319 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.510249    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 20 11:16:04.510444 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.510411    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 20 11:16:04.518691 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.518670    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"]
Jul 20 11:16:04.612763 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.612731    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348fcb7b-6f84-4187-b3de-b50f8462e085-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.612763 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.612764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tcmf\" (UniqueName: \"kubernetes.io/projected/348fcb7b-6f84-4187-b3de-b50f8462e085-kube-api-access-2tcmf\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.613015 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.612840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348fcb7b-6f84-4187-b3de-b50f8462e085-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.613015 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.612902    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348fcb7b-6f84-4187-b3de-b50f8462e085-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.613015 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.612934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348fcb7b-6f84-4187-b3de-b50f8462e085-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.674313 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.674284    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p_4dc732f0-5319-4314-8189-9bb0ba826b4d/storage-initializer/1.log"
Jul 20 11:16:04.674477 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.674397    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"
Jul 20 11:16:04.674477 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.674405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p" event={"ID":"4dc732f0-5319-4314-8189-9bb0ba826b4d","Type":"ContainerDied","Data":"9fb5baa007f811f9252c420ea666f1d205d9ecbdc4ea72f5ed3b9db992b554f7"}
Jul 20 11:16:04.674575 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.674509    2561 scope.go:117] "RemoveContainer" containerID="de243c67f26cf194ff8b333969a66e42b49eb4118d9a8a96ee6ae420b11d8ecc"
Jul 20 11:16:04.709990 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.709962    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"]
Jul 20 11:16:04.713339 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.713316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348fcb7b-6f84-4187-b3de-b50f8462e085-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.713473 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.713349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348fcb7b-6f84-4187-b3de-b50f8462e085-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.713473 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.713382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348fcb7b-6f84-4187-b3de-b50f8462e085-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.713473 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.713404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tcmf\" (UniqueName: \"kubernetes.io/projected/348fcb7b-6f84-4187-b3de-b50f8462e085-kube-api-access-2tcmf\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.713473 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.713448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348fcb7b-6f84-4187-b3de-b50f8462e085-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.713704 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.713671    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-86b65c68d9-ckp4p"]
Jul 20 11:16:04.713915 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.713887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348fcb7b-6f84-4187-b3de-b50f8462e085-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.714092 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.714073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348fcb7b-6f84-4187-b3de-b50f8462e085-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.714092 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.714084    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348fcb7b-6f84-4187-b3de-b50f8462e085-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.716197 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.716177    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348fcb7b-6f84-4187-b3de-b50f8462e085-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.720996 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.720971    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tcmf\" (UniqueName: \"kubernetes.io/projected/348fcb7b-6f84-4187-b3de-b50f8462e085-kube-api-access-2tcmf\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.765458 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.765389    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc732f0-5319-4314-8189-9bb0ba826b4d" path="/var/lib/kubelet/pods/4dc732f0-5319-4314-8189-9bb0ba826b4d/volumes"
Jul 20 11:16:04.819307 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.819282    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:04.949276 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:04.949212    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"]
Jul 20 11:16:04.951576 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:16:04.951551    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348fcb7b_6f84_4187_b3de_b50f8462e085.slice/crio-0a9a82f4369518d1da85e3e39a6f44cbef22e612125622af30eee0f87e198091 WatchSource:0}: Error finding container 0a9a82f4369518d1da85e3e39a6f44cbef22e612125622af30eee0f87e198091: Status 404 returned error can't find the container with id 0a9a82f4369518d1da85e3e39a6f44cbef22e612125622af30eee0f87e198091
Jul 20 11:16:05.680343 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:05.680306    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" event={"ID":"348fcb7b-6f84-4187-b3de-b50f8462e085","Type":"ContainerStarted","Data":"d4b6fcb60aa2b0ba9841cc361b47e8d1518b389554830cfcf0f3f5e27ef5df18"}
Jul 20 11:16:05.680771 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:05.680365    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" event={"ID":"348fcb7b-6f84-4187-b3de-b50f8462e085","Type":"ContainerStarted","Data":"0a9a82f4369518d1da85e3e39a6f44cbef22e612125622af30eee0f87e198091"}
Jul 20 11:16:06.685191 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:06.685154    2561 generic.go:356] "Generic (PLEG): container finished" podID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerID="d4b6fcb60aa2b0ba9841cc361b47e8d1518b389554830cfcf0f3f5e27ef5df18" exitCode=0
Jul 20 11:16:06.685686 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:06.685225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" event={"ID":"348fcb7b-6f84-4187-b3de-b50f8462e085","Type":"ContainerDied","Data":"d4b6fcb60aa2b0ba9841cc361b47e8d1518b389554830cfcf0f3f5e27ef5df18"}
Jul 20 11:16:07.691339 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:07.691305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" event={"ID":"348fcb7b-6f84-4187-b3de-b50f8462e085","Type":"ContainerStarted","Data":"24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9"}
Jul 20 11:16:07.691724 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:07.691360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" event={"ID":"348fcb7b-6f84-4187-b3de-b50f8462e085","Type":"ContainerStarted","Data":"2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8"}
Jul 20 11:16:07.691724 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:07.691537    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:07.711756 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:07.711710    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podStartSLOduration=2.782754836 podStartE2EDuration="3.7116931s" totalImagesPullingTime="0s" totalInitContainerRuntime="928.938264ms" isStatefulPod=false podCreationTimestamp="2026-07-20 11:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:16:07.710570293 +0000 UTC m=+4747.548668879" watchObservedRunningTime="2026-07-20 11:16:07.7116931 +0000 UTC m=+4747.549791684"
Jul 20 11:16:08.695880 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:08.695843    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:08.697342 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:08.697313    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Jul 20 11:16:09.699921 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:09.699877    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Jul 20 11:16:14.705016 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:14.704986    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:16:14.705526 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:14.705500    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Jul 20 11:16:24.705757 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:24.705666    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Jul 20 11:16:34.705820 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:34.705781    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Jul 20 11:16:44.706282 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:44.706238    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Jul 20 11:16:54.705522 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:16:54.705472    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Jul 20 11:17:04.706172 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:04.706130    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Jul 20 11:17:14.705702 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:14.705664    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Jul 20 11:17:16.278605 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:16.278577    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 11:17:16.287180 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:16.287159    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 11:17:22.765951 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:22.765921    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:17:24.546411 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:24.546380    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"]
Jul 20 11:17:24.546824 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:24.546622    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container" containerID="cri-o://2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8" gracePeriod=30
Jul 20 11:17:24.546824 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:24.546683    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kube-rbac-proxy" containerID="cri-o://24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9" gracePeriod=30
Jul 20 11:17:24.700458 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:24.700419    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.70:8643/healthz\": dial tcp 10.132.0.70:8643: connect: connection refused"
Jul 20 11:17:24.978020 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:24.977993    2561 generic.go:356] "Generic (PLEG): container finished" podID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerID="24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9" exitCode=2
Jul 20 11:17:24.978209 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:24.978076    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" event={"ID":"348fcb7b-6f84-4187-b3de-b50f8462e085","Type":"ContainerDied","Data":"24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9"}
Jul 20 11:17:25.607409 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.607372    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"]
Jul 20 11:17:25.610729 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.610706    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:25.613350 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.613330    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 20 11:17:25.613454 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.613368    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 20 11:17:25.622313 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.622290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"]
Jul 20 11:17:25.716170 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.716133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb1e2e0a-7997-4e7d-96af-b2720813d442-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:25.716330 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.716185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb1e2e0a-7997-4e7d-96af-b2720813d442-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:25.716330 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.716205    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb1e2e0a-7997-4e7d-96af-b2720813d442-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:25.716330 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.716257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rsgz\" (UniqueName: \"kubernetes.io/projected/fb1e2e0a-7997-4e7d-96af-b2720813d442-kube-api-access-7rsgz\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:25.817126 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.817094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rsgz\" (UniqueName: \"kubernetes.io/projected/fb1e2e0a-7997-4e7d-96af-b2720813d442-kube-api-access-7rsgz\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:25.817277 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.817147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb1e2e0a-7997-4e7d-96af-b2720813d442-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:25.817277 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.817180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb1e2e0a-7997-4e7d-96af-b2720813d442-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:25.817360 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:17:25.817317    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 20 11:17:25.817360 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.817337    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb1e2e0a-7997-4e7d-96af-b2720813d442-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:25.817426 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:17:25.817383    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1e2e0a-7997-4e7d-96af-b2720813d442-proxy-tls podName:fb1e2e0a-7997-4e7d-96af-b2720813d442 nodeName:}" failed. No retries permitted until 2026-07-20 11:17:26.31736117 +0000 UTC m=+4826.155459738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb1e2e0a-7997-4e7d-96af-b2720813d442-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d" (UID: "fb1e2e0a-7997-4e7d-96af-b2720813d442") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 20 11:17:25.817597 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.817580    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb1e2e0a-7997-4e7d-96af-b2720813d442-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:25.817899 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.817883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb1e2e0a-7997-4e7d-96af-b2720813d442-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:25.825743 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:25.825718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rsgz\" (UniqueName: \"kubernetes.io/projected/fb1e2e0a-7997-4e7d-96af-b2720813d442-kube-api-access-7rsgz\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:26.320628 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:26.320596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb1e2e0a-7997-4e7d-96af-b2720813d442-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:26.322993 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:26.322969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb1e2e0a-7997-4e7d-96af-b2720813d442-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:26.520520 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:26.520486    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:26.647770 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:26.647741    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"]
Jul 20 11:17:26.650964 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:17:26.650936    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1e2e0a_7997_4e7d_96af_b2720813d442.slice/crio-aed9ffd7c964da185eda8b4cbc9f86cd4eef030f40e85582e2a364914b4baada WatchSource:0}: Error finding container aed9ffd7c964da185eda8b4cbc9f86cd4eef030f40e85582e2a364914b4baada: Status 404 returned error can't find the container with id aed9ffd7c964da185eda8b4cbc9f86cd4eef030f40e85582e2a364914b4baada
Jul 20 11:17:26.987104 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:26.987062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d" event={"ID":"fb1e2e0a-7997-4e7d-96af-b2720813d442","Type":"ContainerStarted","Data":"9d57d78f0ab29a29d8eb40575fd7afd3df68b44c49509c52eed1c5009f9e7488"}
Jul 20 11:17:26.987288 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:26.987150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d" event={"ID":"fb1e2e0a-7997-4e7d-96af-b2720813d442","Type":"ContainerStarted","Data":"aed9ffd7c964da185eda8b4cbc9f86cd4eef030f40e85582e2a364914b4baada"}
Jul 20 11:17:28.990250 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:28.990229    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:17:28.995151 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:28.995126    2561 generic.go:356] "Generic (PLEG): container finished" podID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerID="2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8" exitCode=0
Jul 20 11:17:28.995267 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:28.995196    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"
Jul 20 11:17:28.995267 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:28.995204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" event={"ID":"348fcb7b-6f84-4187-b3de-b50f8462e085","Type":"ContainerDied","Data":"2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8"}
Jul 20 11:17:28.995267 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:28.995257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8" event={"ID":"348fcb7b-6f84-4187-b3de-b50f8462e085","Type":"ContainerDied","Data":"0a9a82f4369518d1da85e3e39a6f44cbef22e612125622af30eee0f87e198091"}
Jul 20 11:17:28.995433 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:28.995285    2561 scope.go:117] "RemoveContainer" containerID="24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9"
Jul 20 11:17:29.002794 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.002779    2561 scope.go:117] "RemoveContainer" containerID="2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8"
Jul 20 11:17:29.010734 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.010701    2561 scope.go:117] "RemoveContainer" containerID="d4b6fcb60aa2b0ba9841cc361b47e8d1518b389554830cfcf0f3f5e27ef5df18"
Jul 20 11:17:29.018240 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.018222    2561 scope.go:117] "RemoveContainer" containerID="24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9"
Jul 20 11:17:29.018498 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:17:29.018478    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9\": container with ID starting with 24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9 not found: ID does not exist" containerID="24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9"
Jul 20 11:17:29.018568 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.018506    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9"} err="failed to get container status \"24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9\": rpc error: code = NotFound desc = could not find container \"24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9\": container with ID starting with 24824734d20100d03c9d7d7d98410b9c5d4f9cd7b8783d7c47748e556a1583f9 not found: ID does not exist"
Jul 20 11:17:29.018568 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.018525    2561 scope.go:117] "RemoveContainer" containerID="2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8"
Jul 20 11:17:29.018781 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:17:29.018758    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8\": container with ID starting with 2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8 not found: ID does not exist" containerID="2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8"
Jul 20 11:17:29.018823 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.018791    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8"} err="failed to get container status \"2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8\": rpc error: code = NotFound desc = could not find container \"2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8\": container with ID starting with 2d28ea6769c6b49ccc7eec20d295ac00d34a9554d08b4c90ef36bc62c4b9c3f8 not found: ID does not exist"
Jul 20 11:17:29.018823 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.018813    2561 scope.go:117] "RemoveContainer" containerID="d4b6fcb60aa2b0ba9841cc361b47e8d1518b389554830cfcf0f3f5e27ef5df18"
Jul 20 11:17:29.019071 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:17:29.019052    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b6fcb60aa2b0ba9841cc361b47e8d1518b389554830cfcf0f3f5e27ef5df18\": container with ID starting with d4b6fcb60aa2b0ba9841cc361b47e8d1518b389554830cfcf0f3f5e27ef5df18 not found: ID does not exist" containerID="d4b6fcb60aa2b0ba9841cc361b47e8d1518b389554830cfcf0f3f5e27ef5df18"
Jul 20 11:17:29.019170 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.019073    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b6fcb60aa2b0ba9841cc361b47e8d1518b389554830cfcf0f3f5e27ef5df18"} err="failed to get container status \"d4b6fcb60aa2b0ba9841cc361b47e8d1518b389554830cfcf0f3f5e27ef5df18\": rpc error: code = NotFound desc = could not find container \"d4b6fcb60aa2b0ba9841cc361b47e8d1518b389554830cfcf0f3f5e27ef5df18\": container with ID starting with d4b6fcb60aa2b0ba9841cc361b47e8d1518b389554830cfcf0f3f5e27ef5df18 not found: ID does not exist"
Jul 20 11:17:29.041444 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.041423    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348fcb7b-6f84-4187-b3de-b50f8462e085-kube-rbac-proxy-sar-config\") pod \"348fcb7b-6f84-4187-b3de-b50f8462e085\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") "
Jul 20 11:17:29.041537 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.041451    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348fcb7b-6f84-4187-b3de-b50f8462e085-kserve-provision-location\") pod \"348fcb7b-6f84-4187-b3de-b50f8462e085\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") "
Jul 20 11:17:29.041537 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.041497    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tcmf\" (UniqueName: \"kubernetes.io/projected/348fcb7b-6f84-4187-b3de-b50f8462e085-kube-api-access-2tcmf\") pod \"348fcb7b-6f84-4187-b3de-b50f8462e085\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") "
Jul 20 11:17:29.041537 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.041522    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348fcb7b-6f84-4187-b3de-b50f8462e085-proxy-tls\") pod \"348fcb7b-6f84-4187-b3de-b50f8462e085\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") "
Jul 20 11:17:29.041692 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.041545    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348fcb7b-6f84-4187-b3de-b50f8462e085-cabundle-cert\") pod \"348fcb7b-6f84-4187-b3de-b50f8462e085\" (UID: \"348fcb7b-6f84-4187-b3de-b50f8462e085\") "
Jul 20 11:17:29.041787 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.041763    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348fcb7b-6f84-4187-b3de-b50f8462e085-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "348fcb7b-6f84-4187-b3de-b50f8462e085" (UID: "348fcb7b-6f84-4187-b3de-b50f8462e085"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:17:29.041905 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.041870    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348fcb7b-6f84-4187-b3de-b50f8462e085-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "348fcb7b-6f84-4187-b3de-b50f8462e085" (UID: "348fcb7b-6f84-4187-b3de-b50f8462e085"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:17:29.041997 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.041974    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348fcb7b-6f84-4187-b3de-b50f8462e085-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "348fcb7b-6f84-4187-b3de-b50f8462e085" (UID: "348fcb7b-6f84-4187-b3de-b50f8462e085"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:17:29.043477 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.043457    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348fcb7b-6f84-4187-b3de-b50f8462e085-kube-api-access-2tcmf" (OuterVolumeSpecName: "kube-api-access-2tcmf") pod "348fcb7b-6f84-4187-b3de-b50f8462e085" (UID: "348fcb7b-6f84-4187-b3de-b50f8462e085"). InnerVolumeSpecName "kube-api-access-2tcmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:17:29.043564 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.043492    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348fcb7b-6f84-4187-b3de-b50f8462e085-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "348fcb7b-6f84-4187-b3de-b50f8462e085" (UID: "348fcb7b-6f84-4187-b3de-b50f8462e085"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:17:29.142874 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.142852    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348fcb7b-6f84-4187-b3de-b50f8462e085-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:17:29.142874 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.142873    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348fcb7b-6f84-4187-b3de-b50f8462e085-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:17:29.143020 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.142882    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tcmf\" (UniqueName: \"kubernetes.io/projected/348fcb7b-6f84-4187-b3de-b50f8462e085-kube-api-access-2tcmf\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:17:29.143020 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.142891    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348fcb7b-6f84-4187-b3de-b50f8462e085-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:17:29.143020 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.142899    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348fcb7b-6f84-4187-b3de-b50f8462e085-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:17:29.317447 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.317416    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"]
Jul 20 11:17:29.322681 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:29.322656    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-674bf99d69-brww8"]
Jul 20 11:17:30.765073 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:30.765042    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" path="/var/lib/kubelet/pods/348fcb7b-6f84-4187-b3de-b50f8462e085/volumes"
Jul 20 11:17:32.007360 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:32.007330    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d_fb1e2e0a-7997-4e7d-96af-b2720813d442/storage-initializer/0.log"
Jul 20 11:17:32.007734 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:32.007369    2561 generic.go:356] "Generic (PLEG): container finished" podID="fb1e2e0a-7997-4e7d-96af-b2720813d442" containerID="9d57d78f0ab29a29d8eb40575fd7afd3df68b44c49509c52eed1c5009f9e7488" exitCode=1
Jul 20 11:17:32.007734 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:32.007444    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d" event={"ID":"fb1e2e0a-7997-4e7d-96af-b2720813d442","Type":"ContainerDied","Data":"9d57d78f0ab29a29d8eb40575fd7afd3df68b44c49509c52eed1c5009f9e7488"}
Jul 20 11:17:33.012494 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:33.012467    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d_fb1e2e0a-7997-4e7d-96af-b2720813d442/storage-initializer/0.log"
Jul 20 11:17:33.012906 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:33.012528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d" event={"ID":"fb1e2e0a-7997-4e7d-96af-b2720813d442","Type":"ContainerStarted","Data":"872d210debf217251ba3d4e494e523d45e70aee698efc9bc14dc86e34ea12c76"}
Jul 20 11:17:35.629510 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:35.629463    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"]
Jul 20 11:17:35.629886 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:35.629662    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d" podUID="fb1e2e0a-7997-4e7d-96af-b2720813d442" containerName="storage-initializer" containerID="cri-o://872d210debf217251ba3d4e494e523d45e70aee698efc9bc14dc86e34ea12c76" gracePeriod=30
Jul 20 11:17:36.731396 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.731364    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"]
Jul 20 11:17:36.739812 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.739771    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container"
Jul 20 11:17:36.740000 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.739840    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container"
Jul 20 11:17:36.740060 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.740025    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="storage-initializer"
Jul 20 11:17:36.740060 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.740036    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="storage-initializer"
Jul 20 11:17:36.740060 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.740050    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kube-rbac-proxy"
Jul 20 11:17:36.740060 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.740058    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kube-rbac-proxy"
Jul 20 11:17:36.740286 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.740169    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kube-rbac-proxy"
Jul 20 11:17:36.740286 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.740181    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="348fcb7b-6f84-4187-b3de-b50f8462e085" containerName="kserve-container"
Jul 20 11:17:36.743770 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.743749    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.747330 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.747307    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 11:17:36.747572 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.747536    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 20 11:17:36.747703 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.747316    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 20 11:17:36.750720 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.750698    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"]
Jul 20 11:17:36.797755 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.797721    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.797931 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.797794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.797931 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.797833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pwsx\" (UniqueName: \"kubernetes.io/projected/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kube-api-access-9pwsx\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.797931 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.797858    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.798040 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.797977    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.898416 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.898381    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.898574 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.898440    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.898574 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.898472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pwsx\" (UniqueName: \"kubernetes.io/projected/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kube-api-access-9pwsx\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.898574 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.898494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.898574 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.898545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.899006 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.898964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.899222 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.899199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.899278 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.899259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.901220 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.901199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:36.907073 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:36.907050    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pwsx\" (UniqueName: \"kubernetes.io/projected/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kube-api-access-9pwsx\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:37.027957 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.027889    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d_fb1e2e0a-7997-4e7d-96af-b2720813d442/storage-initializer/1.log"
Jul 20 11:17:37.028309 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.028290    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d_fb1e2e0a-7997-4e7d-96af-b2720813d442/storage-initializer/0.log"
Jul 20 11:17:37.028426 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.028324    2561 generic.go:356] "Generic (PLEG): container finished" podID="fb1e2e0a-7997-4e7d-96af-b2720813d442" containerID="872d210debf217251ba3d4e494e523d45e70aee698efc9bc14dc86e34ea12c76" exitCode=1
Jul 20 11:17:37.028426 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.028402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d" event={"ID":"fb1e2e0a-7997-4e7d-96af-b2720813d442","Type":"ContainerDied","Data":"872d210debf217251ba3d4e494e523d45e70aee698efc9bc14dc86e34ea12c76"}
Jul 20 11:17:37.028515 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.028469    2561 scope.go:117] "RemoveContainer" containerID="9d57d78f0ab29a29d8eb40575fd7afd3df68b44c49509c52eed1c5009f9e7488"
Jul 20 11:17:37.053561 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.053545    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d_fb1e2e0a-7997-4e7d-96af-b2720813d442/storage-initializer/1.log"
Jul 20 11:17:37.057033 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.057018    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:37.057246 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.057233    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:37.099973 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.099949    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb1e2e0a-7997-4e7d-96af-b2720813d442-kube-rbac-proxy-sar-config\") pod \"fb1e2e0a-7997-4e7d-96af-b2720813d442\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") "
Jul 20 11:17:37.100137 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.100041    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rsgz\" (UniqueName: \"kubernetes.io/projected/fb1e2e0a-7997-4e7d-96af-b2720813d442-kube-api-access-7rsgz\") pod \"fb1e2e0a-7997-4e7d-96af-b2720813d442\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") "
Jul 20 11:17:37.100137 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.100078    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb1e2e0a-7997-4e7d-96af-b2720813d442-kserve-provision-location\") pod \"fb1e2e0a-7997-4e7d-96af-b2720813d442\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") "
Jul 20 11:17:37.100137 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.100132    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb1e2e0a-7997-4e7d-96af-b2720813d442-proxy-tls\") pod \"fb1e2e0a-7997-4e7d-96af-b2720813d442\" (UID: \"fb1e2e0a-7997-4e7d-96af-b2720813d442\") "
Jul 20 11:17:37.100349 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.100323    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1e2e0a-7997-4e7d-96af-b2720813d442-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb1e2e0a-7997-4e7d-96af-b2720813d442" (UID: "fb1e2e0a-7997-4e7d-96af-b2720813d442"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:17:37.100406 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.100356    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1e2e0a-7997-4e7d-96af-b2720813d442-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb1e2e0a-7997-4e7d-96af-b2720813d442" (UID: "fb1e2e0a-7997-4e7d-96af-b2720813d442"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:17:37.103221 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.103043    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1e2e0a-7997-4e7d-96af-b2720813d442-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb1e2e0a-7997-4e7d-96af-b2720813d442" (UID: "fb1e2e0a-7997-4e7d-96af-b2720813d442"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:17:37.103221 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.103052    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1e2e0a-7997-4e7d-96af-b2720813d442-kube-api-access-7rsgz" (OuterVolumeSpecName: "kube-api-access-7rsgz") pod "fb1e2e0a-7997-4e7d-96af-b2720813d442" (UID: "fb1e2e0a-7997-4e7d-96af-b2720813d442"). InnerVolumeSpecName "kube-api-access-7rsgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:17:37.183516 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.183442    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"]
Jul 20 11:17:37.185612 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:17:37.185587    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ebf4ae_ffeb_4349_baa2_9716b9a22bef.slice/crio-139919c0b08cb7dd1301837c94793997367936fb8a5b9422c6e3c6b60f0c20b8 WatchSource:0}: Error finding container 139919c0b08cb7dd1301837c94793997367936fb8a5b9422c6e3c6b60f0c20b8: Status 404 returned error can't find the container with id 139919c0b08cb7dd1301837c94793997367936fb8a5b9422c6e3c6b60f0c20b8
Jul 20 11:17:37.201610 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.201589    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rsgz\" (UniqueName: \"kubernetes.io/projected/fb1e2e0a-7997-4e7d-96af-b2720813d442-kube-api-access-7rsgz\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:17:37.201685 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.201613    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb1e2e0a-7997-4e7d-96af-b2720813d442-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:17:37.201685 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.201624    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb1e2e0a-7997-4e7d-96af-b2720813d442-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:17:37.201685 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:37.201634    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb1e2e0a-7997-4e7d-96af-b2720813d442-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:17:38.033247 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:38.033217    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d_fb1e2e0a-7997-4e7d-96af-b2720813d442/storage-initializer/1.log"
Jul 20 11:17:38.033731 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:38.033319    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"
Jul 20 11:17:38.033731 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:38.033320    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d" event={"ID":"fb1e2e0a-7997-4e7d-96af-b2720813d442","Type":"ContainerDied","Data":"aed9ffd7c964da185eda8b4cbc9f86cd4eef030f40e85582e2a364914b4baada"}
Jul 20 11:17:38.033731 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:38.033385    2561 scope.go:117] "RemoveContainer" containerID="872d210debf217251ba3d4e494e523d45e70aee698efc9bc14dc86e34ea12c76"
Jul 20 11:17:38.035287 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:38.035257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" event={"ID":"14ebf4ae-ffeb-4349-baa2-9716b9a22bef","Type":"ContainerStarted","Data":"0df2c05ff9e1a99c4027a4febd422dcaffef20690cdea52662ebca26e36af0ec"}
Jul 20 11:17:38.035384 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:38.035325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" event={"ID":"14ebf4ae-ffeb-4349-baa2-9716b9a22bef","Type":"ContainerStarted","Data":"139919c0b08cb7dd1301837c94793997367936fb8a5b9422c6e3c6b60f0c20b8"}
Jul 20 11:17:38.085603 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:38.085527    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"]
Jul 20 11:17:38.088099 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:38.088074    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-cb5f77477-vft5d"]
Jul 20 11:17:38.765269 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:38.765238    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb1e2e0a-7997-4e7d-96af-b2720813d442" path="/var/lib/kubelet/pods/fb1e2e0a-7997-4e7d-96af-b2720813d442/volumes"
Jul 20 11:17:39.042278 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:39.042180    2561 generic.go:356] "Generic (PLEG): container finished" podID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerID="0df2c05ff9e1a99c4027a4febd422dcaffef20690cdea52662ebca26e36af0ec" exitCode=0
Jul 20 11:17:39.042685 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:39.042274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" event={"ID":"14ebf4ae-ffeb-4349-baa2-9716b9a22bef","Type":"ContainerDied","Data":"0df2c05ff9e1a99c4027a4febd422dcaffef20690cdea52662ebca26e36af0ec"}
Jul 20 11:17:40.048677 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:40.048640    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" event={"ID":"14ebf4ae-ffeb-4349-baa2-9716b9a22bef","Type":"ContainerStarted","Data":"a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1"}
Jul 20 11:17:40.049058 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:40.048707    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:40.049058 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:40.048738    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:40.049058 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:40.048761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" event={"ID":"14ebf4ae-ffeb-4349-baa2-9716b9a22bef","Type":"ContainerStarted","Data":"0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1"}
Jul 20 11:17:40.050067 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:40.050038    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Jul 20 11:17:40.066746 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:40.066703    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podStartSLOduration=3.142546234 podStartE2EDuration="4.066688451s" totalImagesPullingTime="0s" totalInitContainerRuntime="924.142217ms" isStatefulPod=false podCreationTimestamp="2026-07-20 11:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:17:40.065873729 +0000 UTC m=+4839.903972323" watchObservedRunningTime="2026-07-20 11:17:40.066688451 +0000 UTC m=+4839.904787036"
Jul 20 11:17:41.053010 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:41.052967    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Jul 20 11:17:46.057195 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:46.057164    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:17:46.057723 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:46.057663    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Jul 20 11:17:56.057884 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:17:56.057795    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Jul 20 11:18:06.058090 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:06.058050    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Jul 20 11:18:16.058503 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:16.058464    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Jul 20 11:18:26.058386 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:26.058341    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Jul 20 11:18:36.058530 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:36.058482    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Jul 20 11:18:46.057656 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:46.057609    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Jul 20 11:18:56.058276 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:56.058244    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:18:56.769972 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:56.769939    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"]
Jul 20 11:18:56.770206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:56.770167    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container" containerID="cri-o://0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1" gracePeriod=30
Jul 20 11:18:56.770309 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:56.770254    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kube-rbac-proxy" containerID="cri-o://a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1" gracePeriod=30
Jul 20 11:18:57.347368 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.347338    2561 generic.go:356] "Generic (PLEG): container finished" podID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerID="a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1" exitCode=2
Jul 20 11:18:57.347739 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.347408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" event={"ID":"14ebf4ae-ffeb-4349-baa2-9716b9a22bef","Type":"ContainerDied","Data":"a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1"}
Jul 20 11:18:57.828336 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.828302    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"]
Jul 20 11:18:57.828578 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.828566    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1e2e0a-7997-4e7d-96af-b2720813d442" containerName="storage-initializer"
Jul 20 11:18:57.828626 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.828580    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1e2e0a-7997-4e7d-96af-b2720813d442" containerName="storage-initializer"
Jul 20 11:18:57.828626 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.828590    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1e2e0a-7997-4e7d-96af-b2720813d442" containerName="storage-initializer"
Jul 20 11:18:57.828626 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.828595    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1e2e0a-7997-4e7d-96af-b2720813d442" containerName="storage-initializer"
Jul 20 11:18:57.828746 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.828639    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1e2e0a-7997-4e7d-96af-b2720813d442" containerName="storage-initializer"
Jul 20 11:18:57.828746 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.828649    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1e2e0a-7997-4e7d-96af-b2720813d442" containerName="storage-initializer"
Jul 20 11:18:57.831480 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.831463    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:57.834088 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.834046    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 20 11:18:57.834088 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.834078    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 20 11:18:57.842782 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.842761    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"]
Jul 20 11:18:57.886534 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.886512    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:57.886661 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.886554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:57.886661 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.886576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:57.886798 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.886665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-267sl\" (UniqueName: \"kubernetes.io/projected/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kube-api-access-267sl\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:57.987812 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.987774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:57.987985 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.987822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:57.987985 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:18:57.987930    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 20 11:18:57.987985 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.987944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:57.988209 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:18:57.988002    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-proxy-tls podName:fcc51ab6-c69f-40b9-99af-4a911bbe1dc3 nodeName:}" failed. No retries permitted until 2026-07-20 11:18:58.487983268 +0000 UTC m=+4918.326081832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq" (UID: "fcc51ab6-c69f-40b9-99af-4a911bbe1dc3") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 20 11:18:57.988209 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.988032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-267sl\" (UniqueName: \"kubernetes.io/projected/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kube-api-access-267sl\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:57.988331 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.988321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:57.988562 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.988541    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:57.996404 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:57.996384    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-267sl\" (UniqueName: \"kubernetes.io/projected/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kube-api-access-267sl\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:58.492225 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:58.492191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:58.494574 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:58.494548    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:58.741666 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:58.741632    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:18:58.868994 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:58.868963    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"]
Jul 20 11:18:58.872300 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:18:58.872273    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc51ab6_c69f_40b9_99af_4a911bbe1dc3.slice/crio-98a5dba36c0637225556acdd41c5359d0b1e7b38712b001b4a1eb310a5235f9b WatchSource:0}: Error finding container 98a5dba36c0637225556acdd41c5359d0b1e7b38712b001b4a1eb310a5235f9b: Status 404 returned error can't find the container with id 98a5dba36c0637225556acdd41c5359d0b1e7b38712b001b4a1eb310a5235f9b
Jul 20 11:18:59.356614 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:59.356574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq" event={"ID":"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3","Type":"ContainerStarted","Data":"abcd8bbb1846565b0b410191392f2419337def9098af7b04e654e3c410e1984b"}
Jul 20 11:18:59.356819 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:18:59.356634    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq" event={"ID":"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3","Type":"ContainerStarted","Data":"98a5dba36c0637225556acdd41c5359d0b1e7b38712b001b4a1eb310a5235f9b"}
Jul 20 11:19:01.054232 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.054184    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.72:8643/healthz\": dial tcp 10.132.0.72:8643: connect: connection refused"
Jul 20 11:19:01.317983 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.317925    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:19:01.366320 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.366285    2561 generic.go:356] "Generic (PLEG): container finished" podID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerID="0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1" exitCode=0
Jul 20 11:19:01.366492 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.366346    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"
Jul 20 11:19:01.366492 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.366363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" event={"ID":"14ebf4ae-ffeb-4349-baa2-9716b9a22bef","Type":"ContainerDied","Data":"0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1"}
Jul 20 11:19:01.366492 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.366447    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh" event={"ID":"14ebf4ae-ffeb-4349-baa2-9716b9a22bef","Type":"ContainerDied","Data":"139919c0b08cb7dd1301837c94793997367936fb8a5b9422c6e3c6b60f0c20b8"}
Jul 20 11:19:01.366492 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.366474    2561 scope.go:117] "RemoveContainer" containerID="a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1"
Jul 20 11:19:01.374305 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.374288    2561 scope.go:117] "RemoveContainer" containerID="0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1"
Jul 20 11:19:01.381015 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.380992    2561 scope.go:117] "RemoveContainer" containerID="0df2c05ff9e1a99c4027a4febd422dcaffef20690cdea52662ebca26e36af0ec"
Jul 20 11:19:01.387848 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.387821    2561 scope.go:117] "RemoveContainer" containerID="a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1"
Jul 20 11:19:01.388083 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:19:01.388065    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1\": container with ID starting with a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1 not found: ID does not exist" containerID="a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1"
Jul 20 11:19:01.388147 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.388092    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1"} err="failed to get container status \"a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1\": rpc error: code = NotFound desc = could not find container \"a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1\": container with ID starting with a0a32c7fe57125da7834cb481a7f7c44985272e346340a1d1f9a77423cc485e1 not found: ID does not exist"
Jul 20 11:19:01.388147 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.388133    2561 scope.go:117] "RemoveContainer" containerID="0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1"
Jul 20 11:19:01.388359 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:19:01.388344    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1\": container with ID starting with 0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1 not found: ID does not exist" containerID="0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1"
Jul 20 11:19:01.388404 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.388361    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1"} err="failed to get container status \"0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1\": rpc error: code = NotFound desc = could not find container \"0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1\": container with ID starting with 0afbdb140f77a0dcfa42b6ad3e89dd0c3f826c507c62d5e9202fd1db4c1679d1 not found: ID does not exist"
Jul 20 11:19:01.388404 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.388375    2561 scope.go:117] "RemoveContainer" containerID="0df2c05ff9e1a99c4027a4febd422dcaffef20690cdea52662ebca26e36af0ec"
Jul 20 11:19:01.388584 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:19:01.388562    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df2c05ff9e1a99c4027a4febd422dcaffef20690cdea52662ebca26e36af0ec\": container with ID starting with 0df2c05ff9e1a99c4027a4febd422dcaffef20690cdea52662ebca26e36af0ec not found: ID does not exist" containerID="0df2c05ff9e1a99c4027a4febd422dcaffef20690cdea52662ebca26e36af0ec"
Jul 20 11:19:01.388627 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.388592    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df2c05ff9e1a99c4027a4febd422dcaffef20690cdea52662ebca26e36af0ec"} err="failed to get container status \"0df2c05ff9e1a99c4027a4febd422dcaffef20690cdea52662ebca26e36af0ec\": rpc error: code = NotFound desc = could not find container \"0df2c05ff9e1a99c4027a4febd422dcaffef20690cdea52662ebca26e36af0ec\": container with ID starting with 0df2c05ff9e1a99c4027a4febd422dcaffef20690cdea52662ebca26e36af0ec not found: ID does not exist"
Jul 20 11:19:01.413042 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.413020    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-cabundle-cert\") pod \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") "
Jul 20 11:19:01.413185 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.413063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-proxy-tls\") pod \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") "
Jul 20 11:19:01.413229 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.413182    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pwsx\" (UniqueName: \"kubernetes.io/projected/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kube-api-access-9pwsx\") pod \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") "
Jul 20 11:19:01.413268 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.413229    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kube-rbac-proxy-sar-config\") pod \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") "
Jul 20 11:19:01.413268 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.413260    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kserve-provision-location\") pod \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\" (UID: \"14ebf4ae-ffeb-4349-baa2-9716b9a22bef\") "
Jul 20 11:19:01.413436 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.413408    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14ebf4ae-ffeb-4349-baa2-9716b9a22bef" (UID: "14ebf4ae-ffeb-4349-baa2-9716b9a22bef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:19:01.413571 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.413555    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-cabundle-cert\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:19:01.413643 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.413625    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "14ebf4ae-ffeb-4349-baa2-9716b9a22bef" (UID: "14ebf4ae-ffeb-4349-baa2-9716b9a22bef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:19:01.413689 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.413630    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14ebf4ae-ffeb-4349-baa2-9716b9a22bef" (UID: "14ebf4ae-ffeb-4349-baa2-9716b9a22bef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:19:01.415044 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.415025    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14ebf4ae-ffeb-4349-baa2-9716b9a22bef" (UID: "14ebf4ae-ffeb-4349-baa2-9716b9a22bef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:19:01.415315 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.415297    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kube-api-access-9pwsx" (OuterVolumeSpecName: "kube-api-access-9pwsx") pod "14ebf4ae-ffeb-4349-baa2-9716b9a22bef" (UID: "14ebf4ae-ffeb-4349-baa2-9716b9a22bef"). InnerVolumeSpecName "kube-api-access-9pwsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:19:01.513872 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.513837    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pwsx\" (UniqueName: \"kubernetes.io/projected/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kube-api-access-9pwsx\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:19:01.513872 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.513866    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:19:01.513872 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.513875    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:19:01.514140 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.513885    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ebf4ae-ffeb-4349-baa2-9716b9a22bef-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:19:01.688814 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.688780    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"]
Jul 20 11:19:01.692444 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:01.692417    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6b49f44886-ff8qh"]
Jul 20 11:19:02.765868 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:02.765833    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" path="/var/lib/kubelet/pods/14ebf4ae-ffeb-4349-baa2-9716b9a22bef/volumes"
Jul 20 11:19:04.379621 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:04.379591    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq_fcc51ab6-c69f-40b9-99af-4a911bbe1dc3/storage-initializer/0.log"
Jul 20 11:19:04.380006 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:04.379638    2561 generic.go:356] "Generic (PLEG): container finished" podID="fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" containerID="abcd8bbb1846565b0b410191392f2419337def9098af7b04e654e3c410e1984b" exitCode=1
Jul 20 11:19:04.380006 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:04.379715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq" event={"ID":"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3","Type":"ContainerDied","Data":"abcd8bbb1846565b0b410191392f2419337def9098af7b04e654e3c410e1984b"}
Jul 20 11:19:05.384811 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:05.384786    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq_fcc51ab6-c69f-40b9-99af-4a911bbe1dc3/storage-initializer/0.log"
Jul 20 11:19:05.385218 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:05.384887    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq" event={"ID":"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3","Type":"ContainerStarted","Data":"a954df2988c915804cd549042a243b173d628466b919fd4c4bf7a66f20eeccfa"}
Jul 20 11:19:06.390461 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:06.390436    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq_fcc51ab6-c69f-40b9-99af-4a911bbe1dc3/storage-initializer/1.log"
Jul 20 11:19:06.390869 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:06.390785    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq_fcc51ab6-c69f-40b9-99af-4a911bbe1dc3/storage-initializer/0.log"
Jul 20 11:19:06.390869 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:06.390814    2561 generic.go:356] "Generic (PLEG): container finished" podID="fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" containerID="a954df2988c915804cd549042a243b173d628466b919fd4c4bf7a66f20eeccfa" exitCode=1
Jul 20 11:19:06.390869 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:06.390838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq" event={"ID":"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3","Type":"ContainerDied","Data":"a954df2988c915804cd549042a243b173d628466b919fd4c4bf7a66f20eeccfa"}
Jul 20 11:19:06.390976 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:06.390892    2561 scope.go:117] "RemoveContainer" containerID="abcd8bbb1846565b0b410191392f2419337def9098af7b04e654e3c410e1984b"
Jul 20 11:19:06.391244 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:06.391226    2561 scope.go:117] "RemoveContainer" containerID="abcd8bbb1846565b0b410191392f2419337def9098af7b04e654e3c410e1984b"
Jul 20 11:19:06.401034 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:19:06.401008    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq_kserve-ci-e2e-test(fcc51ab6-c69f-40b9-99af-4a911bbe1dc3)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq" podUID="fcc51ab6-c69f-40b9-99af-4a911bbe1dc3"
Jul 20 11:19:06.401187 ip-10-0-134-80 kubenswrapper[2561]: E0720 11:19:06.401165    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq_kserve-ci-e2e-test_fcc51ab6-c69f-40b9-99af-4a911bbe1dc3_0 in pod sandbox 98a5dba36c0637225556acdd41c5359d0b1e7b38712b001b4a1eb310a5235f9b from index: no such id: 'abcd8bbb1846565b0b410191392f2419337def9098af7b04e654e3c410e1984b'" containerID="abcd8bbb1846565b0b410191392f2419337def9098af7b04e654e3c410e1984b"
Jul 20 11:19:06.401238 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:06.401194    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abcd8bbb1846565b0b410191392f2419337def9098af7b04e654e3c410e1984b"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq_kserve-ci-e2e-test_fcc51ab6-c69f-40b9-99af-4a911bbe1dc3_0 in pod sandbox 98a5dba36c0637225556acdd41c5359d0b1e7b38712b001b4a1eb310a5235f9b from index: no such id: 'abcd8bbb1846565b0b410191392f2419337def9098af7b04e654e3c410e1984b'"
Jul 20 11:19:07.396281 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:07.396252    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq_fcc51ab6-c69f-40b9-99af-4a911bbe1dc3/storage-initializer/1.log"
Jul 20 11:19:07.906494 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:07.906434    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"]
Jul 20 11:19:08.032808 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.032786    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq_fcc51ab6-c69f-40b9-99af-4a911bbe1dc3/storage-initializer/1.log"
Jul 20 11:19:08.036059 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.036044    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:19:08.160968 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.160869    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kserve-provision-location\") pod \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") "
Jul 20 11:19:08.160968 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.160928    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-proxy-tls\") pod \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") "
Jul 20 11:19:08.160968 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.160949    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kube-rbac-proxy-sar-config\") pod \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") "
Jul 20 11:19:08.160968 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.160968    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-267sl\" (UniqueName: \"kubernetes.io/projected/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kube-api-access-267sl\") pod \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\" (UID: \"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3\") "
Jul 20 11:19:08.161266 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.161224    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" (UID: "fcc51ab6-c69f-40b9-99af-4a911bbe1dc3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:19:08.161336 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.161317    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" (UID: "fcc51ab6-c69f-40b9-99af-4a911bbe1dc3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:19:08.162992 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.162966    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" (UID: "fcc51ab6-c69f-40b9-99af-4a911bbe1dc3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:19:08.163146 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.163011    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kube-api-access-267sl" (OuterVolumeSpecName: "kube-api-access-267sl") pod "fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" (UID: "fcc51ab6-c69f-40b9-99af-4a911bbe1dc3"). InnerVolumeSpecName "kube-api-access-267sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:19:08.262349 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.262303    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kserve-provision-location\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:19:08.262349 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.262342    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-proxy-tls\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:19:08.262349 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.262351    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:19:08.262349 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.262360    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-267sl\" (UniqueName: \"kubernetes.io/projected/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3-kube-api-access-267sl\") on node \"ip-10-0-134-80.ec2.internal\" DevicePath \"\""
Jul 20 11:19:08.400781 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.400754    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq_fcc51ab6-c69f-40b9-99af-4a911bbe1dc3/storage-initializer/1.log"
Jul 20 11:19:08.401213 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.400799    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq" event={"ID":"fcc51ab6-c69f-40b9-99af-4a911bbe1dc3","Type":"ContainerDied","Data":"98a5dba36c0637225556acdd41c5359d0b1e7b38712b001b4a1eb310a5235f9b"}
Jul 20 11:19:08.401213 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.400847    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"
Jul 20 11:19:08.401213 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.400858    2561 scope.go:117] "RemoveContainer" containerID="a954df2988c915804cd549042a243b173d628466b919fd4c4bf7a66f20eeccfa"
Jul 20 11:19:08.434096 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.433985    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"]
Jul 20 11:19:08.436583 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.436558    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bdc8789fb-8gxlq"]
Jul 20 11:19:08.766297 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:08.766208    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" path="/var/lib/kubelet/pods/fcc51ab6-c69f-40b9-99af-4a911bbe1dc3/volumes"
Jul 20 11:19:43.354932 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.354841    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p54z8/must-gather-p8jcw"]
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355220    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" containerName="storage-initializer"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355239    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" containerName="storage-initializer"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355255    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="storage-initializer"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355263    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="storage-initializer"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355278    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kube-rbac-proxy"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355287    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kube-rbac-proxy"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355301    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355308    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355387    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kube-rbac-proxy"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355399    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ebf4ae-ffeb-4349-baa2-9716b9a22bef" containerName="kserve-container"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355408    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" containerName="storage-initializer"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355416    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" containerName="storage-initializer"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355483    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" containerName="storage-initializer"
Jul 20 11:19:43.355612 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.355492    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc51ab6-c69f-40b9-99af-4a911bbe1dc3" containerName="storage-initializer"
Jul 20 11:19:43.358656 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.358635    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p54z8/must-gather-p8jcw"
Jul 20 11:19:43.361053 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.361023    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p54z8\"/\"openshift-service-ca.crt\""
Jul 20 11:19:43.361176 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.361051    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p54z8\"/\"default-dockercfg-5bcv8\""
Jul 20 11:19:43.362147 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.362129    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p54z8\"/\"kube-root-ca.crt\""
Jul 20 11:19:43.366279 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.366250    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p54z8/must-gather-p8jcw"]
Jul 20 11:19:43.507369 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.507328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scwbs\" (UniqueName: \"kubernetes.io/projected/86ab585f-5d2d-43e7-95af-cff129b4a438-kube-api-access-scwbs\") pod \"must-gather-p8jcw\" (UID: \"86ab585f-5d2d-43e7-95af-cff129b4a438\") " pod="openshift-must-gather-p54z8/must-gather-p8jcw"
Jul 20 11:19:43.507546 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.507411    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/86ab585f-5d2d-43e7-95af-cff129b4a438-must-gather-output\") pod \"must-gather-p8jcw\" (UID: \"86ab585f-5d2d-43e7-95af-cff129b4a438\") " pod="openshift-must-gather-p54z8/must-gather-p8jcw"
Jul 20 11:19:43.608157 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.608057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/86ab585f-5d2d-43e7-95af-cff129b4a438-must-gather-output\") pod \"must-gather-p8jcw\" (UID: \"86ab585f-5d2d-43e7-95af-cff129b4a438\") " pod="openshift-must-gather-p54z8/must-gather-p8jcw"
Jul 20 11:19:43.608157 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.608129    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scwbs\" (UniqueName: \"kubernetes.io/projected/86ab585f-5d2d-43e7-95af-cff129b4a438-kube-api-access-scwbs\") pod \"must-gather-p8jcw\" (UID: \"86ab585f-5d2d-43e7-95af-cff129b4a438\") " pod="openshift-must-gather-p54z8/must-gather-p8jcw"
Jul 20 11:19:43.608396 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.608379    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/86ab585f-5d2d-43e7-95af-cff129b4a438-must-gather-output\") pod \"must-gather-p8jcw\" (UID: \"86ab585f-5d2d-43e7-95af-cff129b4a438\") " pod="openshift-must-gather-p54z8/must-gather-p8jcw"
Jul 20 11:19:43.616155 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.616125    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scwbs\" (UniqueName: \"kubernetes.io/projected/86ab585f-5d2d-43e7-95af-cff129b4a438-kube-api-access-scwbs\") pod \"must-gather-p8jcw\" (UID: \"86ab585f-5d2d-43e7-95af-cff129b4a438\") " pod="openshift-must-gather-p54z8/must-gather-p8jcw"
Jul 20 11:19:43.667762 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.667732    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p54z8/must-gather-p8jcw"
Jul 20 11:19:43.789862 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.789836    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p54z8/must-gather-p8jcw"]
Jul 20 11:19:43.792226 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:19:43.792190    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86ab585f_5d2d_43e7_95af_cff129b4a438.slice/crio-d6b80d4cd841398f6fa63493b2a685a2bd94360f099473df908f8a4100e8b339 WatchSource:0}: Error finding container d6b80d4cd841398f6fa63493b2a685a2bd94360f099473df908f8a4100e8b339: Status 404 returned error can't find the container with id d6b80d4cd841398f6fa63493b2a685a2bd94360f099473df908f8a4100e8b339
Jul 20 11:19:43.793867 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:43.793851    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 11:19:44.535892 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:44.535858    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p54z8/must-gather-p8jcw" event={"ID":"86ab585f-5d2d-43e7-95af-cff129b4a438","Type":"ContainerStarted","Data":"d6b80d4cd841398f6fa63493b2a685a2bd94360f099473df908f8a4100e8b339"}
Jul 20 11:19:45.543206 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:45.543162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p54z8/must-gather-p8jcw" event={"ID":"86ab585f-5d2d-43e7-95af-cff129b4a438","Type":"ContainerStarted","Data":"8b366437992e00a53b3a04a055bc5f26b004d0fd6d3b2729435682559fad134d"}
Jul 20 11:19:45.543686 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:45.543217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p54z8/must-gather-p8jcw" event={"ID":"86ab585f-5d2d-43e7-95af-cff129b4a438","Type":"ContainerStarted","Data":"aa85c75166190c42280a59d60587a94c6b144ebfa180437ebab35193fce063d2"}
Jul 20 11:19:45.561481 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:45.561414    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p54z8/must-gather-p8jcw" podStartSLOduration=1.6773899399999999 podStartE2EDuration="2.561391593s" totalImagesPullingTime="884.001653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:19:45.558974998 +0000 UTC m=+4965.397073585" watchObservedRunningTime="2026-07-20 11:19:45.561391593 +0000 UTC m=+4965.399490179"
Jul 20 11:19:46.180423 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:46.180380    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rw9rr_3ab13120-6b0f-4805-bc85-bb1ee7384987/global-pull-secret-syncer/0.log"
Jul 20 11:19:46.273148 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:46.273100    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kj6v2_013cc0d4-fe49-43cd-9fb5-1041e0b0001d/konnectivity-agent/0.log"
Jul 20 11:19:46.351531 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:46.351494    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-80.ec2.internal_d8acf0f9821033101209753c1ff5ad61/haproxy/0.log"
Jul 20 11:19:50.580794 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:50.580768    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqkhq_5c096253-3ee3-424e-bbfe-40d8360c181a/node-exporter/0.log"
Jul 20 11:19:50.602506 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:50.602483    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqkhq_5c096253-3ee3-424e-bbfe-40d8360c181a/kube-rbac-proxy/0.log"
Jul 20 11:19:50.620684 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:50.620659    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqkhq_5c096253-3ee3-424e-bbfe-40d8360c181a/init-textfile/0.log"
Jul 20 11:19:54.730676 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:54.730638    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"]
Jul 20 11:19:54.736043 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:54.736018    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:54.744680 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:54.743768    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"]
Jul 20 11:19:54.904974 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:54.904935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-proc\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:54.905157 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:54.905054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn674\" (UniqueName: \"kubernetes.io/projected/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-kube-api-access-dn674\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:54.905157 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:54.905086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-podres\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:54.905157 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:54.905144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-sys\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:54.905278 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:54.905192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-lib-modules\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.006063 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.005971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-sys\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.006063 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.006020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-lib-modules\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.006063 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.006046    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-sys\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.006353 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.006069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-proc\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.006353 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.006148    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn674\" (UniqueName: \"kubernetes.io/projected/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-kube-api-access-dn674\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.006353 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.006182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-lib-modules\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.006353 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.006182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-podres\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.006353 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.006218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-proc\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.006353 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.006273    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-podres\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.014287 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.014260    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn674\" (UniqueName: \"kubernetes.io/projected/0db69fef-dbf8-4bb8-aa83-7c19cb4f3037-kube-api-access-dn674\") pod \"perf-node-gather-daemonset-7hbhb\" (UID: \"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037\") " pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.042798 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.042774    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lx7wv_71be3c38-364e-4ace-8643-1324c5176e0d/dns/0.log"
Jul 20 11:19:55.049080 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.049055    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.062267 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.062245    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lx7wv_71be3c38-364e-4ace-8643-1324c5176e0d/kube-rbac-proxy/0.log"
Jul 20 11:19:55.164306 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.164283    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mn8bv_6a052b9a-0d1c-4b01-8b75-1f435ec83841/dns-node-resolver/0.log"
Jul 20 11:19:55.198965 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.198344    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"]
Jul 20 11:19:55.201285 ip-10-0-134-80 kubenswrapper[2561]: W0720 11:19:55.201249    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0db69fef_dbf8_4bb8_aa83_7c19cb4f3037.slice/crio-e2ae4388ac1b645d7e79760baffe406d6e638c0576d900b63b00600682bea01e WatchSource:0}: Error finding container e2ae4388ac1b645d7e79760baffe406d6e638c0576d900b63b00600682bea01e: Status 404 returned error can't find the container with id e2ae4388ac1b645d7e79760baffe406d6e638c0576d900b63b00600682bea01e
Jul 20 11:19:55.594611 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.594524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb" event={"ID":"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037","Type":"ContainerStarted","Data":"0b8a8ee77c285327251ed66a8d5cbf134ac2e16518bbb1cc7358da33fb743387"}
Jul 20 11:19:55.594611 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.594596    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb" event={"ID":"0db69fef-dbf8-4bb8-aa83-7c19cb4f3037","Type":"ContainerStarted","Data":"e2ae4388ac1b645d7e79760baffe406d6e638c0576d900b63b00600682bea01e"}
Jul 20 11:19:55.595370 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.595340    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:19:55.612318 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.612272    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb" podStartSLOduration=1.612258653 podStartE2EDuration="1.612258653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 11:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:19:55.611649818 +0000 UTC m=+4975.449748403" watchObservedRunningTime="2026-07-20 11:19:55.612258653 +0000 UTC m=+4975.450357238"
Jul 20 11:19:55.807909 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:55.807881    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n5hmv_33f1203a-dd9a-46be-96bd-47184ae3fccf/node-ca/0.log"
Jul 20 11:19:56.822168 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:56.822142    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bzw96_dcc4c661-d4f8-44b5-8536-5d91d0c7d5a5/serve-healthcheck-canary/0.log"
Jul 20 11:19:57.228762 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:57.228738    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ngrqc_ce4ca6d7-070c-410b-8c44-ccc8b089bca8/kube-rbac-proxy/0.log"
Jul 20 11:19:57.248043 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:57.248020    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ngrqc_ce4ca6d7-070c-410b-8c44-ccc8b089bca8/exporter/0.log"
Jul 20 11:19:57.265597 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:57.265553    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ngrqc_ce4ca6d7-070c-410b-8c44-ccc8b089bca8/extractor/0.log"
Jul 20 11:19:59.507407 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:59.507367    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-645f4f9567-w4qrj_5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263/manager/1.log"
Jul 20 11:19:59.569007 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:59.568978    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-645f4f9567-w4qrj_5bdeee7a-715e-4ca0-b0a3-9a55a6cc8263/manager/0.log"
Jul 20 11:19:59.614024 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:59.613995    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-zdxq7_38f8e412-4a70-4059-9712-45d0ada332db/server/0.log"
Jul 20 11:19:59.760713 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:59.760596    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-qjmq5_7d5831d2-0e10-454b-9fb5-fb909c11e317/manager/1.log"
Jul 20 11:19:59.907482 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:59.907453    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-qjmq5_7d5831d2-0e10-454b-9fb5-fb909c11e317/manager/0.log"
Jul 20 11:19:59.936355 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:59.936327    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2cpq9_a1a02fda-409f-4d26-90d4-ba4a0eb041a2/s3-init/0.log"
Jul 20 11:19:59.950864 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:59.950838    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2cpq9_a1a02fda-409f-4d26-90d4-ba4a0eb041a2/download-hf-model/0.log"
Jul 20 11:19:59.976014 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:59.975993    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6qztd_c811e518-aeb6-4346-bfd7-4b68ce20ba48/s3-tls-init-custom/0.log"
Jul 20 11:19:59.991102 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:19:59.991079    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6qztd_c811e518-aeb6-4346-bfd7-4b68ce20ba48/download-hf-model/0.log"
Jul 20 11:20:00.016977 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:00.016925    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-kttst_93836ca5-dbd2-421f-8c04-a14b44d0da8e/s3-tls-init-serving/0.log"
Jul 20 11:20:00.032171 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:00.032150    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-kttst_93836ca5-dbd2-421f-8c04-a14b44d0da8e/download-hf-model/0.log"
Jul 20 11:20:00.062032 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:00.062012    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-pdprk_d1890299-2569-489f-9d25-a4a1b6786292/seaweedfs/0.log"
Jul 20 11:20:00.108558 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:00.108524    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-ggh99_d6823c38-491b-43b7-9bde-50ea7cec69f8/seaweedfs-tls-serving/0.log"
Jul 20 11:20:02.615876 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:02.615839    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p54z8/perf-node-gather-daemonset-7hbhb"
Jul 20 11:20:05.353652 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:05.353624    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8s76_102e4f9f-8d5c-40d6-8b00-7f1a13888508/kube-multus-additional-cni-plugins/0.log"
Jul 20 11:20:05.373048 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:05.373023    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8s76_102e4f9f-8d5c-40d6-8b00-7f1a13888508/egress-router-binary-copy/0.log"
Jul 20 11:20:05.396077 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:05.396052    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8s76_102e4f9f-8d5c-40d6-8b00-7f1a13888508/cni-plugins/0.log"
Jul 20 11:20:05.416129 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:05.416094    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8s76_102e4f9f-8d5c-40d6-8b00-7f1a13888508/bond-cni-plugin/0.log"
Jul 20 11:20:05.440476 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:05.440448    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8s76_102e4f9f-8d5c-40d6-8b00-7f1a13888508/routeoverride-cni/0.log"
Jul 20 11:20:05.459532 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:05.459502    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8s76_102e4f9f-8d5c-40d6-8b00-7f1a13888508/whereabouts-cni-bincopy/0.log"
Jul 20 11:20:05.478644 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:05.478614    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8s76_102e4f9f-8d5c-40d6-8b00-7f1a13888508/whereabouts-cni/0.log"
Jul 20 11:20:05.538644 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:05.538617    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l8xwg_0787b82a-7c63-40a4-af72-91cfa5bfd7e4/kube-multus/0.log"
Jul 20 11:20:05.674972 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:05.674942    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n4x2n_df44767e-e243-4004-a43e-4d2982ddd197/network-metrics-daemon/0.log"
Jul 20 11:20:05.693095 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:05.693075    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n4x2n_df44767e-e243-4004-a43e-4d2982ddd197/kube-rbac-proxy/0.log"
Jul 20 11:20:06.458992 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:06.458954    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-controller/0.log"
Jul 20 11:20:06.472652 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:06.472630    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/1.log"
Jul 20 11:20:06.472781 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:06.472621    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovn-acl-logging/0.log"
Jul 20 11:20:06.488327 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:06.488302    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/kube-rbac-proxy-node/0.log"
Jul 20 11:20:06.506404 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:06.506380    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/kube-rbac-proxy-ovn-metrics/0.log"
Jul 20 11:20:06.522938 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:06.522914    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/northd/0.log"
Jul 20 11:20:06.539874 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:06.539856    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/nbdb/0.log"
Jul 20 11:20:06.557396 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:06.557370    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/sbdb/0.log"
Jul 20 11:20:06.608015 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:06.607987    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxc68_b1fd10ce-1d5a-4844-81d4-ef2e75cea222/ovnkube-controller/0.log"
Jul 20 11:20:07.947696 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:07.947664    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-fcpp5_ffd754c6-58d4-429e-aa73-9e3cd0f1626e/network-check-target-container/0.log"
Jul 20 11:20:08.826212 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:08.826186    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2wmxz_c840916c-ce29-4f39-8f28-b2c462411b72/iptables-alerter/0.log"
Jul 20 11:20:09.469839 ip-10-0-134-80 kubenswrapper[2561]: I0720 11:20:09.469807    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-qsb9p_8d85828e-89be-41f7-a3e6-9011e4280223/tuned/0.log"